[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:44:10 dist-test-slave-kghz krb5kdc[400](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:44:10 dist-test-slave-kghz krb5kdc[400](info): set up 2 sockets
Sep 25 04:44:10 dist-test-slave-kghz krb5kdc[400](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260925 04:44:11.555377 393 mini_kdc.cc:121] Time spent starting KDC: real 1.637s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/krb5kdc/test-user.keytab.
Sep 25 04:44:11 dist-test-slave-kghz krb5kdc[400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311451, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:44:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:44:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260925 04:44:11.685691 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:38919
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:41575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:38919
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:44:11.799271 413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:44:11.799619 413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:44:11.799702 413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:44:11.803885 413 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:44:11.803987 413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:44:11.804025 413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:44:11.804086 413 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:44:11.804128 413 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:44:11.809532 413 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:41575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:38919
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:38919
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.413
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:44:11.810796 413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:44:11.811995 413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:44:11.821808 419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:11.821758 418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:11.821791 421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:11.821874 413 server_base.cc:1034] running on GCE node
I20260925 04:44:11.822819 413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:44:11.824064 413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:44:11.825245 413 hybrid_clock.cc:648] HybridClock initialized: now 1790311451825187 us; error 71 us; skew 500 ppm
Sep 25 04:44:11 dist-test-slave-kghz krb5kdc[400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311451, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:44:11.828315 413 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260925 04:44:11.829406 413 webserver.cc:460] Webserver started at http://127.0.98.126:40715/ using document root <none> and password file <none>
I20260925 04:44:11.830055 413 fs_manager.cc:359] Metadata directory not provided
I20260925 04:44:11.830125 413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:44:11.830391 413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:44:11.832628 413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/minicluster-data/master-0/data/instance:
uuid: "36ba6aa2911a4f7b807093f9df897e6c"
format_stamp: "Formatted at 2026-09-25 04:44:11 on dist-test-slave-kghz"
server_key: "642dbce9629f7349b99ad7d4d9bc495b"
server_key_iv: "ff65ace05f0038eb9d668a43f37e3778"
server_key_version: "encryptionkey@0"
I20260925 04:44:11.833266 413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/minicluster-data/master-0/wal/instance:
uuid: "36ba6aa2911a4f7b807093f9df897e6c"
format_stamp: "Formatted at 2026-09-25 04:44:11 on dist-test-slave-kghz"
server_key: "642dbce9629f7349b99ad7d4d9bc495b"
server_key_iv: "ff65ace05f0038eb9d668a43f37e3778"
server_key_version: "encryptionkey@0"
I20260925 04:44:11.837534 413 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 04:44:11.840248 428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:11.841616 413 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 04:44:11.841761 413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/minicluster-data/master-0/wal
uuid: "36ba6aa2911a4f7b807093f9df897e6c"
format_stamp: "Formatted at 2026-09-25 04:44:11 on dist-test-slave-kghz"
server_key: "642dbce9629f7349b99ad7d4d9bc495b"
server_key_iv: "ff65ace05f0038eb9d668a43f37e3778"
server_key_version: "encryptionkey@0"
I20260925 04:44:11.841881 413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:44:11.863967 413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:44:11.867503 413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:44:11.867758 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:44:11.877384 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:38919
I20260925 04:44:11.877662 480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:38919 every 8 connection(s)
I20260925 04:44:11.878614 413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
I20260925 04:44:11.881713 481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:44:11.882831 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 413
I20260925 04:44:11.882980 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/minicluster-data/master-0/wal/instance
I20260925 04:44:11.884286 393 external_mini_cluster.cc:1428] Setting key 4e0796c348b5596393b0fdfef3966371
I20260925 04:44:11.888281 481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36ba6aa2911a4f7b807093f9df897e6c: Bootstrap starting.
I20260925 04:44:11.890841 481 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36ba6aa2911a4f7b807093f9df897e6c: Neither blocks nor log segments found. Creating new log.
I20260925 04:44:11.891595 481 log.cc:824] T 00000000000000000000000000000000 P 36ba6aa2911a4f7b807093f9df897e6c: Log is configured to *not* fsync() on all Append() calls
I20260925 04:44:11.893502 481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36ba6aa2911a4f7b807093f9df897e6c: No bootstrap required, opened a new log
I20260925 04:44:11.897037 481 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36ba6aa2911a4f7b807093f9df897e6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36ba6aa2911a4f7b807093f9df897e6c" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 38919 } }
I20260925 04:44:11.897315 481 raft_consensus.cc:374] T 00000000000000000000000000000000 P 36ba6aa2911a4f7b807093f9df897e6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:44:11.897400 481 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36ba6aa2911a4f7b807093f9df897e6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36ba6aa2911a4f7b807093f9df897e6c, State: Initialized, Role: FOLLOWER
I20260925 04:44:11.897921 481 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36ba6aa2911a4f7b807093f9df897e6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36ba6aa2911a4f7b807093f9df897e6c" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 38919 } }
I20260925 04:44:11.898082 481 raft_consensus.cc:388] T 00000000000000000000000000000000 P 36ba6aa2911a4f7b807093f9df897e6c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:44:11.898171 481 raft_consensus.cc:482] T 00000000000000000000000000000000 P 36ba6aa2911a4f7b807093f9df897e6c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:44:11.898295 481 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36ba6aa2911a4f7b807093f9df897e6c [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:44:11.899269 481 raft_consensus.cc:504] T 00000000000000000000000000000000 P 36ba6aa2911a4f7b807093f9df897e6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36ba6aa2911a4f7b807093f9df897e6c" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 38919 } }
I20260925 04:44:11.899655 481 leader_election.cc:302] T 00000000000000000000000000000000 P 36ba6aa2911a4f7b807093f9df897e6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36ba6aa2911a4f7b807093f9df897e6c; no voters:
I20260925 04:44:11.900022 481 leader_election.cc:288] T 00000000000000000000000000000000 P 36ba6aa2911a4f7b807093f9df897e6c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:44:11.900101 486 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 36ba6aa2911a4f7b807093f9df897e6c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:44:11.900341 486 raft_consensus.cc:686] T 00000000000000000000000000000000 P 36ba6aa2911a4f7b807093f9df897e6c [term 1 LEADER]: Becoming Leader. State: Replica: 36ba6aa2911a4f7b807093f9df897e6c, State: Running, Role: LEADER
I20260925 04:44:11.900669 486 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36ba6aa2911a4f7b807093f9df897e6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36ba6aa2911a4f7b807093f9df897e6c" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 38919 } }
I20260925 04:44:11.901443 481 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36ba6aa2911a4f7b807093f9df897e6c [sys.catalog]: configured and running, proceeding with master startup.
Sep 25 04:44:11 dist-test-slave-kghz krb5kdc[400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311451, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:44:11.902419 487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36ba6aa2911a4f7b807093f9df897e6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "36ba6aa2911a4f7b807093f9df897e6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36ba6aa2911a4f7b807093f9df897e6c" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 38919 } } }
I20260925 04:44:11.902549 487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36ba6aa2911a4f7b807093f9df897e6c [sys.catalog]: This master's current role is: LEADER
I20260925 04:44:11.902963 495 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:44:11.903749 488 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36ba6aa2911a4f7b807093f9df897e6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36ba6aa2911a4f7b807093f9df897e6c. Latest consensus state: current_term: 1 leader_uuid: "36ba6aa2911a4f7b807093f9df897e6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36ba6aa2911a4f7b807093f9df897e6c" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 38919 } } }
I20260925 04:44:11.904388 488 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36ba6aa2911a4f7b807093f9df897e6c [sys.catalog]: This master's current role is: LEADER
I20260925 04:44:11.906798 495 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:44:11.911300 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:11.887592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36126 (local address 127.0.98.126:38919)
0925 04:44:11.888032 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:11.888048 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:11.889145 (+ 1097us) server_negotiation.cc:407] Connection header received
0925 04:44:11.892486 (+ 3341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:11.892503 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:11.892822 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:11.893165 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:11.895467 (+ 2302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:11.896451 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:11.897827 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:11.898487 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:11.898666 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:11.904606 (+ 5940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:11.904613 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:11.904631 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:11.904668 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:11.906848 (+ 2180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:11.907688 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:11.907695 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:11.907700 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:11.907843 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:11.908271 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:11.908274 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:11.908276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:11.908817 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:11.908994 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:11.909772 (+ 778us) server_negotiation.cc:299] Negotiation successful
0925 04:44:11.910078 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":138,"threads_started":1}
I20260925 04:44:11.914832 495 catalog_manager.cc:1347] Generated new cluster ID: a99851468260462cb55e55685bbf5fe6
I20260925 04:44:11.914917 495 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:44:11.930634 495 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:44:11.931605 495 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:44:11.942929 495 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 36ba6aa2911a4f7b807093f9df897e6c: Generated new TSK 0
I20260925 04:44:11.943758 495 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/krb5kdc/kudu_127.0.98.65.keytab.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/krb5kdc/kudu_127.0.98.65.keytab.
I20260925 04:44:11.979391 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:0
--local_ip_for_outbound_sockets=127.0.98.65
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:38919
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:41575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:44:12.094003 507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:44:12.094389 507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:44:12.094444 507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:44:12.098585 507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:44:12.098659 507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:44:12.098722 507 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260925 04:44:12.104511 507 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:41575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:38919
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:44:12.105667 507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:44:12.106729 507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:44:12.116832 512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:12.116837 513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:12.116989 507 server_base.cc:1034] running on GCE node
W20260925 04:44:12.116832 515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:12.117585 507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:44:12.118335 507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:44:12.119520 507 hybrid_clock.cc:648] HybridClock initialized: now 1790311452119503 us; error 29 us; skew 500 ppm
Sep 25 04:44:12 dist-test-slave-kghz krb5kdc[400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311452, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:44:12.122390 507 init.cc:377] Logged in from keytab as kudu/127.0.98.65@KRBTEST.COM (short username kudu)
I20260925 04:44:12.123315 507 webserver.cc:460] Webserver started at http://127.0.98.65:44557/ using document root <none> and password file <none>
I20260925 04:44:12.123875 507 fs_manager.cc:359] Metadata directory not provided
I20260925 04:44:12.123936 507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:44:12.124176 507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:44:12.126358 507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/minicluster-data/ts-0/data/instance:
uuid: "641c7b55ae8740e88448e20a9e6ef697"
format_stamp: "Formatted at 2026-09-25 04:44:12 on dist-test-slave-kghz"
server_key: "a400ceaca4a60e1d5226b34cb161431c"
server_key_iv: "514b77ee4011960b80fbec6b78ab4f5c"
server_key_version: "encryptionkey@0"
I20260925 04:44:12.126943 507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/minicluster-data/ts-0/wal/instance:
uuid: "641c7b55ae8740e88448e20a9e6ef697"
format_stamp: "Formatted at 2026-09-25 04:44:12 on dist-test-slave-kghz"
server_key: "a400ceaca4a60e1d5226b34cb161431c"
server_key_iv: "514b77ee4011960b80fbec6b78ab4f5c"
server_key_version: "encryptionkey@0"
I20260925 04:44:12.130739 507 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260925 04:44:12.133164 522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:12.134626 507 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:44:12.134749 507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/minicluster-data/ts-0/wal
uuid: "641c7b55ae8740e88448e20a9e6ef697"
format_stamp: "Formatted at 2026-09-25 04:44:12 on dist-test-slave-kghz"
server_key: "a400ceaca4a60e1d5226b34cb161431c"
server_key_iv: "514b77ee4011960b80fbec6b78ab4f5c"
server_key_version: "encryptionkey@0"
I20260925 04:44:12.134862 507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:44:12.149204 507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:44:12.152417 507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:44:12.152626 507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:44:12.153239 507 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:44:12.154209 507 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:44:12.154315 507 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:12.154402 507 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:44:12.154452 507 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:12.163281 507 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:34847
I20260925 04:44:12.163300 635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:34847 every 8 connection(s)
I20260925 04:44:12.164215 507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
I20260925 04:44:12.165836 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 507
I20260925 04:44:12.165959 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/minicluster-data/ts-0/wal/instance
I20260925 04:44:12.166235 393 external_mini_cluster.cc:1428] Setting key 8e2ae4868e8c2437780c99669b4b6936
Sep 25 04:44:12 dist-test-slave-kghz krb5kdc[400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311452, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.66@KRBTEST.COM" created.
I20260925 04:44:12.179801 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:12.166140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:57763 (local address 127.0.98.126:38919)
0925 04:44:12.166316 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:12.166321 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:12.167360 (+ 1039us) server_negotiation.cc:407] Connection header received
0925 04:44:12.168860 (+ 1500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:12.168866 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:12.168927 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:12.169022 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:12.170286 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:12.170830 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:12.172005 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:12.172235 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:12.172368 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:12.175348 (+ 2980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:12.175351 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:12.175353 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:12.175376 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:12.177194 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:12.177782 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:12.177785 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:12.177786 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:12.177869 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:12.178305 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:12.178310 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:12.178311 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:12.178540 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:12.178670 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:12.179335 (+ 665us) server_negotiation.cc:299] Negotiation successful
0925 04:44:12.179513 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 04:44:12.180502 638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:12.166654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38919 (local address 127.0.98.65:57763)
0925 04:44:12.167184 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:12.167245 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:12.168634 (+ 1389us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:12.169182 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:12.169213 (+ 31us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:12.169598 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:12.170102 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:12.170118 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:12.170963 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:12.170967 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:12.171862 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:12.171870 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:12.172424 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:12.172428 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:12.172602 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:12.173293 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:12.173313 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:12.175104 (+ 1791us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:44:12.177339 (+ 2235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:12.177356 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:12.177371 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:12.177654 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:12.177962 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:12.177964 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:12.177966 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:12.178109 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:12.178691 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:44:12.178701 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:44:12.179046 (+ 345us) client_negotiation.cc:769] Sending connection context
0925 04:44:12.179232 (+ 186us) client_negotiation.cc:240] Negotiation successful
0925 04:44:12.179534 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":143,"threads_started":1}
I20260925 04:44:12.181944 636 heartbeater.cc:347] Connected to a master server at 127.0.98.126:38919
I20260925 04:44:12.182305 636 heartbeater.cc:464] Registering TS with master...
I20260925 04:44:12.182868 636 heartbeater.cc:511] Master 127.0.98.126:38919 requested a full tablet report, sending...
I20260925 04:44:12.184840 444 ts_manager.cc:194] Registered new tserver with Master: 641c7b55ae8740e88448e20a9e6ef697 (127.0.98.65:34847)
I20260925 04:44:12.186657 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.65@KRBTEST.COM'} at 127.0.98.65:57763
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/krb5kdc/kudu_127.0.98.66.keytab.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/krb5kdc/kudu_127.0.98.66.keytab.
I20260925 04:44:12.197794 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:0
--local_ip_for_outbound_sockets=127.0.98.66
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:38919
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:41575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:44:12.310501 641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:44:12.310840 641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:44:12.310920 641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:44:12.315246 641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:44:12.315332 641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:44:12.315436 641 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260925 04:44:12.320942 641 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:41575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:38919
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.641
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:44:12.322074 641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:44:12.323220 641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:44:12.333009 647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:12.333004 649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:12.333072 646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:12.333274 641 server_base.cc:1034] running on GCE node
I20260925 04:44:12.333755 641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:44:12.334494 641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:44:12.335650 641 hybrid_clock.cc:648] HybridClock initialized: now 1790311452335616 us; error 50 us; skew 500 ppm
Sep 25 04:44:12 dist-test-slave-kghz krb5kdc[400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311452, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:44:12.338563 641 init.cc:377] Logged in from keytab as kudu/127.0.98.66@KRBTEST.COM (short username kudu)
I20260925 04:44:12.339555 641 webserver.cc:460] Webserver started at http://127.0.98.66:37293/ using document root <none> and password file <none>
I20260925 04:44:12.340129 641 fs_manager.cc:359] Metadata directory not provided
I20260925 04:44:12.340188 641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:44:12.340411 641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:44:12.342583 641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/minicluster-data/ts-1/data/instance:
uuid: "e0c528d2c8924b858170b6d3c5044987"
format_stamp: "Formatted at 2026-09-25 04:44:12 on dist-test-slave-kghz"
server_key: "0b8a30d3f10ee9c7a6dfeb1e61132dac"
server_key_iv: "02aca363275a2b0bb7843eed37f64417"
server_key_version: "encryptionkey@0"
I20260925 04:44:12.343161 641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/minicluster-data/ts-1/wal/instance:
uuid: "e0c528d2c8924b858170b6d3c5044987"
format_stamp: "Formatted at 2026-09-25 04:44:12 on dist-test-slave-kghz"
server_key: "0b8a30d3f10ee9c7a6dfeb1e61132dac"
server_key_iv: "02aca363275a2b0bb7843eed37f64417"
server_key_version: "encryptionkey@0"
I20260925 04:44:12.347199 641 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260925 04:44:12.349802 656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:12.351226 641 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 04:44:12.351394 641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/minicluster-data/ts-1/wal
uuid: "e0c528d2c8924b858170b6d3c5044987"
format_stamp: "Formatted at 2026-09-25 04:44:12 on dist-test-slave-kghz"
server_key: "0b8a30d3f10ee9c7a6dfeb1e61132dac"
server_key_iv: "02aca363275a2b0bb7843eed37f64417"
server_key_version: "encryptionkey@0"
I20260925 04:44:12.351527 641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:44:12.365872 641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:44:12.369007 641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:44:12.369231 641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:44:12.369871 641 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:44:12.370766 641 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:44:12.370841 641 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:12.370908 641 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:44:12.370952 641 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:12.382027 641 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:35379
I20260925 04:44:12.382055 769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:35379 every 8 connection(s)
I20260925 04:44:12.383152 641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
I20260925 04:44:12.384238 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 641
I20260925 04:44:12.384392 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/minicluster-data/ts-1/wal/instance
I20260925 04:44:12.384661 393 external_mini_cluster.cc:1428] Setting key 21a01af9db24c3ed8cf5c1344b390786
WARNING: no policy specified for kudu/127.0.98.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.67@KRBTEST.COM" created.
Sep 25 04:44:12 dist-test-slave-kghz krb5kdc[400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311452, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:44:12.405671 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:12.385628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:53081 (local address 127.0.98.126:38919)
0925 04:44:12.385766 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:12.385770 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:12.386765 (+ 995us) server_negotiation.cc:407] Connection header received
0925 04:44:12.388911 (+ 2146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:12.388916 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:12.388986 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:12.389097 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:12.390955 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:12.391589 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:12.393612 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:12.393915 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:12.394010 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:12.400226 (+ 6216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:12.400230 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:12.400232 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:12.400265 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:12.402513 (+ 2248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:12.403292 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:12.403299 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:12.403304 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:12.403414 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:12.404120 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:12.404143 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:12.404145 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:12.404459 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:12.404587 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:12.405219 (+ 632us) server_negotiation.cc:299] Negotiation successful
0925 04:44:12.405401 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 04:44:12.406649 772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:12.385961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38919 (local address 127.0.98.66:53081)
0925 04:44:12.386584 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:12.386656 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:12.388644 (+ 1988us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:12.389286 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:12.389294 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:12.389914 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:12.390741 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:12.390768 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:12.391745 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:12.391752 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:12.393461 (+ 1709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:12.393474 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:12.394098 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:12.394104 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:12.394382 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:12.395081 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:12.395104 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:12.399940 (+ 4836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:44:12.402757 (+ 2817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:12.402764 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:12.402787 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:12.403140 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:12.403548 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:12.403584 (+ 36us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:12.403587 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:12.403969 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:12.404592 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:44:12.404602 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:44:12.404918 (+ 316us) client_negotiation.cc:769] Sending connection context
0925 04:44:12.405103 (+ 185us) client_negotiation.cc:240] Negotiation successful
0925 04:44:12.405531 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":150,"threads_started":1}
I20260925 04:44:12.408034 770 heartbeater.cc:347] Connected to a master server at 127.0.98.126:38919
I20260925 04:44:12.408355 770 heartbeater.cc:464] Registering TS with master...
I20260925 04:44:12.408962 770 heartbeater.cc:511] Master 127.0.98.126:38919 requested a full tablet report, sending...
I20260925 04:44:12.410175 444 ts_manager.cc:194] Registered new tserver with Master: e0c528d2c8924b858170b6d3c5044987 (127.0.98.66:35379)
I20260925 04:44:12.410912 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.66@KRBTEST.COM'} at 127.0.98.66:53081
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/krb5kdc/kudu_127.0.98.67.keytab.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/krb5kdc/kudu_127.0.98.67.keytab.
I20260925 04:44:12.418205 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:0
--local_ip_for_outbound_sockets=127.0.98.67
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:38919
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:41575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:44:12.551563 775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:44:12.551855 775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:44:12.551903 775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:44:12.556809 775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:44:12.556895 775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:44:12.556990 775 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260925 04:44:12.563091 775 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:41575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:38919
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.775
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:44:12.564147 775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:44:12.565127 775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:44:12.574702 783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:12.574723 780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:12.574724 781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:12.574959 775 server_base.cc:1034] running on GCE node
I20260925 04:44:12.575434 775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:44:12.576109 775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:44:12.577325 775 hybrid_clock.cc:648] HybridClock initialized: now 1790311452577309 us; error 29 us; skew 500 ppm
Sep 25 04:44:12 dist-test-slave-kghz krb5kdc[400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311452, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:44:12.580309 775 init.cc:377] Logged in from keytab as kudu/127.0.98.67@KRBTEST.COM (short username kudu)
I20260925 04:44:12.581278 775 webserver.cc:460] Webserver started at http://127.0.98.67:35201/ using document root <none> and password file <none>
I20260925 04:44:12.581864 775 fs_manager.cc:359] Metadata directory not provided
I20260925 04:44:12.581954 775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:44:12.582190 775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:44:12.584408 775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/minicluster-data/ts-2/data/instance:
uuid: "ad0ee0b0f0cd4b9296ccf2f0cb20a806"
format_stamp: "Formatted at 2026-09-25 04:44:12 on dist-test-slave-kghz"
server_key: "9f498922ee3c644af01d9d34e559b851"
server_key_iv: "5534a6ea2da577b20d40d91eb0c88d51"
server_key_version: "encryptionkey@0"
I20260925 04:44:12.584995 775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/minicluster-data/ts-2/wal/instance:
uuid: "ad0ee0b0f0cd4b9296ccf2f0cb20a806"
format_stamp: "Formatted at 2026-09-25 04:44:12 on dist-test-slave-kghz"
server_key: "9f498922ee3c644af01d9d34e559b851"
server_key_iv: "5534a6ea2da577b20d40d91eb0c88d51"
server_key_version: "encryptionkey@0"
I20260925 04:44:12.588945 775 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260925 04:44:12.591696 790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:12.593039 775 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 04:44:12.593204 775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/minicluster-data/ts-2/wal
uuid: "ad0ee0b0f0cd4b9296ccf2f0cb20a806"
format_stamp: "Formatted at 2026-09-25 04:44:12 on dist-test-slave-kghz"
server_key: "9f498922ee3c644af01d9d34e559b851"
server_key_iv: "5534a6ea2da577b20d40d91eb0c88d51"
server_key_version: "encryptionkey@0"
I20260925 04:44:12.593333 775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:44:12.624895 775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:44:12.628638 775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:44:12.628868 775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:44:12.629558 775 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:44:12.630545 775 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:44:12.630612 775 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:12.630679 775 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:44:12.630734 775 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:12.641525 775 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:40477
I20260925 04:44:12.641549 903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:40477 every 8 connection(s)
I20260925 04:44:12.642534 775 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
I20260925 04:44:12.645170 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 775
I20260925 04:44:12.645351 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790311449873301-393-0/minicluster-data/ts-2/wal/instance
I20260925 04:44:12.645732 393 external_mini_cluster.cc:1428] Setting key b563a308c4164e60da37b71ecf73927b
Sep 25 04:44:12 dist-test-slave-kghz krb5kdc[400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311452, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:44:12.657735 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:12.644499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:40335 (local address 127.0.98.126:38919)
0925 04:44:12.644649 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:12.644654 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:12.645654 (+ 1000us) server_negotiation.cc:407] Connection header received
0925 04:44:12.647577 (+ 1923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:12.647580 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:12.647637 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:12.647745 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:12.649152 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:12.649690 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:12.650956 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:12.651177 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:12.651287 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:12.653905 (+ 2618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:12.653908 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:12.653911 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:12.653937 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:12.655562 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:12.656119 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:12.656125 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:12.656127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:12.656214 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:12.656562 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:12.656567 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:12.656569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:12.656794 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:12.656906 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:12.657440 (+ 534us) server_negotiation.cc:299] Negotiation successful
0925 04:44:12.657565 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 04:44:12.658699 906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:12.644983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38919 (local address 127.0.98.67:40335)
0925 04:44:12.645488 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:12.645545 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:12.647331 (+ 1786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:12.647948 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:12.647956 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:12.648478 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:12.649002 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:12.649016 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:12.649819 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:12.649822 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:12.650810 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:12.650819 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:12.651328 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:12.651332 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:12.651517 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:12.652091 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:12.652108 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:12.653679 (+ 1571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:44:12.655701 (+ 2022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:12.655715 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:12.655729 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:12.655999 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:12.656312 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:12.656316 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:12.656319 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:12.656463 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:12.656902 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:44:12.656910 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:44:12.657192 (+ 282us) client_negotiation.cc:769] Sending connection context
0925 04:44:12.657401 (+ 209us) client_negotiation.cc:240] Negotiation successful
0925 04:44:12.657743 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":138,"threads_started":1}
I20260925 04:44:12.660578 904 heartbeater.cc:347] Connected to a master server at 127.0.98.126:38919
I20260925 04:44:12.660851 904 heartbeater.cc:464] Registering TS with master...
I20260925 04:44:12.661425 904 heartbeater.cc:511] Master 127.0.98.126:38919 requested a full tablet report, sending...
I20260925 04:44:12.662521 444 ts_manager.cc:194] Registered new tserver with Master: ad0ee0b0f0cd4b9296ccf2f0cb20a806 (127.0.98.67:40477)
I20260925 04:44:12.663117 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.67@KRBTEST.COM'} at 127.0.98.67:40335
I20260925 04:44:12.673110 393 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 04:44:12 dist-test-slave-kghz krb5kdc[400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311452, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 04:44:12 dist-test-slave-kghz krb5kdc[400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311452, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.65@KRBTEST.COM
I20260925 04:44:12.702098 911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:12.690425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49502 (local address 127.0.98.65:34847)
0925 04:44:12.690788 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:12.690797 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:12.690823 (+ 26us) server_negotiation.cc:407] Connection header received
0925 04:44:12.690877 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:12.690883 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:12.691095 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:12.691227 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:12.692008 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:12.692825 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:12.693894 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:12.694429 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:12.694531 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:12.697953 (+ 3422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:12.697964 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:12.698000 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:12.698043 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:12.700008 (+ 1965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:12.700588 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:12.700594 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:12.700595 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:12.700684 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:12.700947 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:12.700949 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:12.700951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:12.701202 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:12.701384 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:12.701682 (+ 298us) server_negotiation.cc:299] Negotiation successful
0925 04:44:12.701902 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":105,"threads_started":1}
W20260925 04:44:12.703485 550 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49502
Sep 25 04:44:12 dist-test-slave-kghz krb5kdc[400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311452, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 04:44:12 dist-test-slave-kghz krb5kdc[400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311452, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.65@KRBTEST.COM
I20260925 04:44:12.730628 911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:12.721656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49516 (local address 127.0.98.65:34847)
0925 04:44:12.721848 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:12.721852 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:12.721957 (+ 105us) server_negotiation.cc:407] Connection header received
0925 04:44:12.722068 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:12.722070 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:12.722115 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:12.722188 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:12.722993 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:12.723512 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:12.724336 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:12.724563 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:12.724648 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:12.727279 (+ 2631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:12.727283 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:12.727285 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:12.727311 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:12.728854 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:12.729246 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:12.729249 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:12.729250 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:12.729390 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:12.729645 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:12.729648 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:12.729649 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:12.729829 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:12.729948 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:12.730310 (+ 362us) server_negotiation.cc:299] Negotiation successful
0925 04:44:12.730430 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260925 04:44:12.731796 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 507
I20260925 04:44:12.738322 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 641
I20260925 04:44:12.746133 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 775
I20260925 04:44:12.751907 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 413
2026-09-25T04:44:12Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2866 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:44:12 dist-test-slave-kghz krb5kdc[918](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:44:12 dist-test-slave-kghz krb5kdc[918](info): set up 2 sockets
Sep 25 04:44:12 dist-test-slave-kghz krb5kdc[918](info): commencing operation
krb5kdc: starting...
W20260925 04:44:13.826473 393 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/krb5kdc/test-user.keytab.
Sep 25 04:44:13 dist-test-slave-kghz krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:44:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:44:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260925 04:44:13.958187 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:43625
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:45167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:43625
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:44:14.072122 931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:44:14.072418 931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:44:14.072461 931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:44:14.076490 931 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:44:14.076556 931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:44:14.076575 931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:44:14.076598 931 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:44:14.076614 931 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:44:14.081986 931 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:45167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:43625
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:43625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:44:14.083097 931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:44:14.084056 931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:44:14.092664 936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:14.092653 937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:14.092729 931 server_base.cc:1034] running on GCE node
W20260925 04:44:14.092653 939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:14.093428 931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:44:14.094717 931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:44:14.095927 931 hybrid_clock.cc:648] HybridClock initialized: now 1790311454095910 us; error 32 us; skew 500 ppm
Sep 25 04:44:14 dist-test-slave-kghz krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311454, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:44:14.098826 931 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260925 04:44:14.099680 931 webserver.cc:460] Webserver started at http://127.0.98.126:38437/ using document root <none> and password file <none>
I20260925 04:44:14.100237 931 fs_manager.cc:359] Metadata directory not provided
I20260925 04:44:14.100310 931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:44:14.100512 931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:44:14.102658 931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/minicluster-data/master-0/data/instance:
uuid: "93d9ba0aed9f44b498003ffc81f3a01e"
format_stamp: "Formatted at 2026-09-25 04:44:14 on dist-test-slave-kghz"
server_key: "fe5c83613a2d014106a2cb5aa355c3df"
server_key_iv: "f8533375007441246ed3a33674fb7c2a"
server_key_version: "encryptionkey@0"
I20260925 04:44:14.103204 931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/minicluster-data/master-0/wal/instance:
uuid: "93d9ba0aed9f44b498003ffc81f3a01e"
format_stamp: "Formatted at 2026-09-25 04:44:14 on dist-test-slave-kghz"
server_key: "fe5c83613a2d014106a2cb5aa355c3df"
server_key_iv: "f8533375007441246ed3a33674fb7c2a"
server_key_version: "encryptionkey@0"
I20260925 04:44:14.107245 931 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 04:44:14.110051 946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:14.111572 931 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 04:44:14.111699 931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/minicluster-data/master-0/wal
uuid: "93d9ba0aed9f44b498003ffc81f3a01e"
format_stamp: "Formatted at 2026-09-25 04:44:14 on dist-test-slave-kghz"
server_key: "fe5c83613a2d014106a2cb5aa355c3df"
server_key_iv: "f8533375007441246ed3a33674fb7c2a"
server_key_version: "encryptionkey@0"
I20260925 04:44:14.111804 931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:44:14.124493 931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:44:14.127782 931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:44:14.127938 931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:44:14.135128 931 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:43625
I20260925 04:44:14.135149 998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:43625 every 8 connection(s)
I20260925 04:44:14.136103 931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
I20260925 04:44:14.139133 999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:44:14.144845 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 931
I20260925 04:44:14.144972 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/minicluster-data/master-0/wal/instance
I20260925 04:44:14.145047 999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93d9ba0aed9f44b498003ffc81f3a01e: Bootstrap starting.
I20260925 04:44:14.145236 393 external_mini_cluster.cc:1428] Setting key d476a94b10072b6b2c88e170897fe9f5
I20260925 04:44:14.147778 999 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93d9ba0aed9f44b498003ffc81f3a01e: Neither blocks nor log segments found. Creating new log.
I20260925 04:44:14.148571 999 log.cc:824] T 00000000000000000000000000000000 P 93d9ba0aed9f44b498003ffc81f3a01e: Log is configured to *not* fsync() on all Append() calls
I20260925 04:44:14.150589 999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93d9ba0aed9f44b498003ffc81f3a01e: No bootstrap required, opened a new log
I20260925 04:44:14.154036 999 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93d9ba0aed9f44b498003ffc81f3a01e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93d9ba0aed9f44b498003ffc81f3a01e" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 43625 } }
I20260925 04:44:14.154332 999 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93d9ba0aed9f44b498003ffc81f3a01e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:44:14.154419 999 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93d9ba0aed9f44b498003ffc81f3a01e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93d9ba0aed9f44b498003ffc81f3a01e, State: Initialized, Role: FOLLOWER
Sep 25 04:44:14 dist-test-slave-kghz krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:44:14.154985 999 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93d9ba0aed9f44b498003ffc81f3a01e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93d9ba0aed9f44b498003ffc81f3a01e" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 43625 } }
I20260925 04:44:14.155146 999 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93d9ba0aed9f44b498003ffc81f3a01e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:44:14.155237 999 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93d9ba0aed9f44b498003ffc81f3a01e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:44:14.155371 999 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93d9ba0aed9f44b498003ffc81f3a01e [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:44:14.156603 999 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93d9ba0aed9f44b498003ffc81f3a01e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93d9ba0aed9f44b498003ffc81f3a01e" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 43625 } }
I20260925 04:44:14.156982 999 leader_election.cc:302] T 00000000000000000000000000000000 P 93d9ba0aed9f44b498003ffc81f3a01e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93d9ba0aed9f44b498003ffc81f3a01e; no voters:
I20260925 04:44:14.157364 999 leader_election.cc:288] T 00000000000000000000000000000000 P 93d9ba0aed9f44b498003ffc81f3a01e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:44:14.157447 1004 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93d9ba0aed9f44b498003ffc81f3a01e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:44:14.157642 1004 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93d9ba0aed9f44b498003ffc81f3a01e [term 1 LEADER]: Becoming Leader. State: Replica: 93d9ba0aed9f44b498003ffc81f3a01e, State: Running, Role: LEADER
I20260925 04:44:14.157954 1004 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93d9ba0aed9f44b498003ffc81f3a01e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93d9ba0aed9f44b498003ffc81f3a01e" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 43625 } }
I20260925 04:44:14.158774 999 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93d9ba0aed9f44b498003ffc81f3a01e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:44:14.159642 1006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93d9ba0aed9f44b498003ffc81f3a01e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93d9ba0aed9f44b498003ffc81f3a01e. Latest consensus state: current_term: 1 leader_uuid: "93d9ba0aed9f44b498003ffc81f3a01e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93d9ba0aed9f44b498003ffc81f3a01e" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 43625 } } }
I20260925 04:44:14.159773 1006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93d9ba0aed9f44b498003ffc81f3a01e [sys.catalog]: This master's current role is: LEADER
I20260925 04:44:14.160108 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93d9ba0aed9f44b498003ffc81f3a01e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "93d9ba0aed9f44b498003ffc81f3a01e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93d9ba0aed9f44b498003ffc81f3a01e" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 43625 } } }
I20260925 04:44:14.160181 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93d9ba0aed9f44b498003ffc81f3a01e [sys.catalog]: This master's current role is: LEADER
I20260925 04:44:14.160434 1013 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:44:14.160946 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:14.146790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50760 (local address 127.0.98.126:43625)
0925 04:44:14.147282 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:14.147299 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:14.147336 (+ 37us) server_negotiation.cc:407] Connection header received
0925 04:44:14.148194 (+ 858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:14.148214 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:14.148578 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:14.148969 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:14.149767 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:14.150908 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:14.151783 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:14.152271 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:14.152447 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:14.155138 (+ 2691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:14.155144 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:14.155163 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:14.155197 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:14.157270 (+ 2073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:14.157819 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:14.157826 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:14.157832 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:14.157972 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:14.158329 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:14.158332 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:14.158334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:14.158745 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:14.158983 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:14.159518 (+ 535us) server_negotiation.cc:299] Negotiation successful
0925 04:44:14.159860 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":139,"threads_started":1}
I20260925 04:44:14.164237 1013 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:44:14.170514 1013 catalog_manager.cc:1347] Generated new cluster ID: 0a628ac6ef734ee88dc53c07ec8e43ae
I20260925 04:44:14.170605 1013 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:44:14.196352 1013 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:44:14.197525 1013 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:44:14.204463 1013 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 93d9ba0aed9f44b498003ffc81f3a01e: Generated new TSK 0
I20260925 04:44:14.205369 1013 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/krb5kdc/kudu_127.0.98.65.keytab.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/krb5kdc/kudu_127.0.98.65.keytab.
I20260925 04:44:14.246726 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:0
--local_ip_for_outbound_sockets=127.0.98.65
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:43625
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:45167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:44:14.368788 1025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:44:14.369068 1025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:44:14.369114 1025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:44:14.373153 1025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:44:14.373219 1025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:44:14.373282 1025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260925 04:44:14.378706 1025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:45167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:43625
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:44:14.379941 1025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:44:14.380954 1025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:44:14.391268 1030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:14.391338 1033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:14.391304 1025 server_base.cc:1034] running on GCE node
W20260925 04:44:14.391316 1031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:14.392226 1025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:44:14.392977 1025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:44:14.394141 1025 hybrid_clock.cc:648] HybridClock initialized: now 1790311454394118 us; error 41 us; skew 500 ppm
Sep 25 04:44:14 dist-test-slave-kghz krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311454, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:44:14.397071 1025 init.cc:377] Logged in from keytab as kudu/127.0.98.65@KRBTEST.COM (short username kudu)
I20260925 04:44:14.398015 1025 webserver.cc:460] Webserver started at http://127.0.98.65:41109/ using document root <none> and password file <none>
I20260925 04:44:14.398662 1025 fs_manager.cc:359] Metadata directory not provided
I20260925 04:44:14.398726 1025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:44:14.398947 1025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:44:14.401036 1025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/minicluster-data/ts-0/data/instance:
uuid: "9292326bc9e844d8a0a50c159b30fdcb"
format_stamp: "Formatted at 2026-09-25 04:44:14 on dist-test-slave-kghz"
server_key: "b2e4dc5f91fd7bb7eed6831c83e870ef"
server_key_iv: "0627a4d983487e4912c80722a8c7cfd3"
server_key_version: "encryptionkey@0"
I20260925 04:44:14.401608 1025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/minicluster-data/ts-0/wal/instance:
uuid: "9292326bc9e844d8a0a50c159b30fdcb"
format_stamp: "Formatted at 2026-09-25 04:44:14 on dist-test-slave-kghz"
server_key: "b2e4dc5f91fd7bb7eed6831c83e870ef"
server_key_iv: "0627a4d983487e4912c80722a8c7cfd3"
server_key_version: "encryptionkey@0"
I20260925 04:44:14.405490 1025 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 04:44:14.408108 1040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:14.409446 1025 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 04:44:14.409584 1025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/minicluster-data/ts-0/wal
uuid: "9292326bc9e844d8a0a50c159b30fdcb"
format_stamp: "Formatted at 2026-09-25 04:44:14 on dist-test-slave-kghz"
server_key: "b2e4dc5f91fd7bb7eed6831c83e870ef"
server_key_iv: "0627a4d983487e4912c80722a8c7cfd3"
server_key_version: "encryptionkey@0"
I20260925 04:44:14.409700 1025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:44:14.447569 1025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:44:14.451258 1025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:44:14.451479 1025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:44:14.452111 1025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:44:14.452979 1025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:44:14.453053 1025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:14.453128 1025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:44:14.453171 1025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:14.462661 1025 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:42257
I20260925 04:44:14.462707 1153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:42257 every 8 connection(s)
I20260925 04:44:14.463636 1025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
I20260925 04:44:14.464046 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 1025
I20260925 04:44:14.464211 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/minicluster-data/ts-0/wal/instance
I20260925 04:44:14.464512 393 external_mini_cluster.cc:1428] Setting key 98cef675bbd7519dc4fca936a9c25ac5
WARNING: no policy specified for kudu/127.0.98.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.66@KRBTEST.COM" created.
Sep 25 04:44:14 dist-test-slave-kghz krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311454, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:44:14.483153 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:14.465668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:47279 (local address 127.0.98.126:43625)
0925 04:44:14.465817 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:14.465821 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:14.466953 (+ 1132us) server_negotiation.cc:407] Connection header received
0925 04:44:14.468572 (+ 1619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:14.468575 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:14.468629 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:14.468711 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:14.470436 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:14.470980 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:14.472965 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:14.473213 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:14.473312 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:14.479139 (+ 5827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:14.479142 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:14.479144 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:14.479246 (+ 102us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:14.480875 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:14.481463 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:14.481477 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:14.481479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:14.481563 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:14.481929 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:14.481932 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:14.481934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:14.482130 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:14.482238 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:14.482844 (+ 606us) server_negotiation.cc:299] Negotiation successful
0925 04:44:14.482975 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 04:44:14.484002 1156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:14.466155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:43625 (local address 127.0.98.65:47279)
0925 04:44:14.466762 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:14.466828 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:14.468355 (+ 1527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:14.468893 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:14.468905 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:14.469469 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:14.470231 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:14.470277 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:14.471133 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:14.471137 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:14.472788 (+ 1651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:14.472805 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:14.473399 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:14.473407 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:14.473654 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:14.474407 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:14.474429 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:14.478910 (+ 4481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:44:14.480998 (+ 2088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:14.481004 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:14.481020 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:14.481334 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:14.481667 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:14.481673 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:14.481679 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:14.481820 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:14.482297 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:44:14.482305 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:44:14.482566 (+ 261us) client_negotiation.cc:769] Sending connection context
0925 04:44:14.482762 (+ 196us) client_negotiation.cc:240] Negotiation successful
0925 04:44:14.483121 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":132,"threads_started":1}
I20260925 04:44:14.485323 1154 heartbeater.cc:347] Connected to a master server at 127.0.98.126:43625
I20260925 04:44:14.485591 1154 heartbeater.cc:464] Registering TS with master...
I20260925 04:44:14.486114 1154 heartbeater.cc:511] Master 127.0.98.126:43625 requested a full tablet report, sending...
I20260925 04:44:14.487702 963 ts_manager.cc:194] Registered new tserver with Master: 9292326bc9e844d8a0a50c159b30fdcb (127.0.98.65:42257)
I20260925 04:44:14.488912 963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.65@KRBTEST.COM'} at 127.0.98.65:47279
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/krb5kdc/kudu_127.0.98.66.keytab.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/krb5kdc/kudu_127.0.98.66.keytab.
I20260925 04:44:14.493877 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:0
--local_ip_for_outbound_sockets=127.0.98.66
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:43625
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:45167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:44:14.607436 1159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:44:14.607721 1159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:44:14.607767 1159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:44:14.612042 1159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:44:14.612110 1159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:44:14.612178 1159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260925 04:44:14.617753 1159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:45167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:43625
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:44:14.618912 1159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:44:14.619915 1159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:44:14.630056 1167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:14.630054 1165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:14.630052 1164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:14.630398 1159 server_base.cc:1034] running on GCE node
I20260925 04:44:14.630972 1159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:44:14.631701 1159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:44:14.632882 1159 hybrid_clock.cc:648] HybridClock initialized: now 1790311454632836 us; error 53 us; skew 500 ppm
Sep 25 04:44:14 dist-test-slave-kghz krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311454, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:44:14.635969 1159 init.cc:377] Logged in from keytab as kudu/127.0.98.66@KRBTEST.COM (short username kudu)
I20260925 04:44:14.637159 1159 webserver.cc:460] Webserver started at http://127.0.98.66:37809/ using document root <none> and password file <none>
I20260925 04:44:14.637975 1159 fs_manager.cc:359] Metadata directory not provided
I20260925 04:44:14.638100 1159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:44:14.638402 1159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:44:14.641047 1159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/minicluster-data/ts-1/data/instance:
uuid: "e4210df3a8744694aa3920849c4bfad9"
format_stamp: "Formatted at 2026-09-25 04:44:14 on dist-test-slave-kghz"
server_key: "a73ee8ee5090592ca9a2ea2599deec8d"
server_key_iv: "19081345f482c1c5067a437143390cc8"
server_key_version: "encryptionkey@0"
I20260925 04:44:14.641582 1159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/minicluster-data/ts-1/wal/instance:
uuid: "e4210df3a8744694aa3920849c4bfad9"
format_stamp: "Formatted at 2026-09-25 04:44:14 on dist-test-slave-kghz"
server_key: "a73ee8ee5090592ca9a2ea2599deec8d"
server_key_iv: "19081345f482c1c5067a437143390cc8"
server_key_version: "encryptionkey@0"
I20260925 04:44:14.645659 1159 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 04:44:14.648422 1174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:14.649832 1159 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 04:44:14.649977 1159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/minicluster-data/ts-1/wal
uuid: "e4210df3a8744694aa3920849c4bfad9"
format_stamp: "Formatted at 2026-09-25 04:44:14 on dist-test-slave-kghz"
server_key: "a73ee8ee5090592ca9a2ea2599deec8d"
server_key_iv: "19081345f482c1c5067a437143390cc8"
server_key_version: "encryptionkey@0"
I20260925 04:44:14.650100 1159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:44:14.663367 1159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:44:14.666824 1159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:44:14.667032 1159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:44:14.667654 1159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:44:14.668493 1159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:44:14.668540 1159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:14.668581 1159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:44:14.668635 1159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:14.678032 1159 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:43673
I20260925 04:44:14.678051 1287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:43673 every 8 connection(s)
I20260925 04:44:14.679026 1159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
I20260925 04:44:14.680205 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 1159
I20260925 04:44:14.680315 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/minicluster-data/ts-1/wal/instance
I20260925 04:44:14.680590 393 external_mini_cluster.cc:1428] Setting key 8d14c2c47aba73068388c00fb3f4c6a7
WARNING: no policy specified for kudu/127.0.98.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.67@KRBTEST.COM" created.
Sep 25 04:44:14 dist-test-slave-kghz krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311454, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:44:14.697837 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:14.681094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:48645 (local address 127.0.98.126:43625)
0925 04:44:14.681224 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:14.681231 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:14.682281 (+ 1050us) server_negotiation.cc:407] Connection header received
0925 04:44:14.683770 (+ 1489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:14.683773 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:14.683831 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:14.683905 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:14.685433 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:14.686447 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:14.687651 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:14.688040 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:14.688168 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:14.693715 (+ 5547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:14.693718 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:14.693720 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:14.693746 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:14.695336 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:14.695981 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:14.695987 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:14.695992 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:14.696091 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:14.696465 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:14.696472 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:14.696476 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:14.696787 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:14.696902 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:14.697421 (+ 519us) server_negotiation.cc:299] Negotiation successful
0925 04:44:14.697576 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 04:44:14.698685 1290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:14.681568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:43625 (local address 127.0.98.66:48645)
0925 04:44:14.682111 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:14.682155 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:14.683524 (+ 1369us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:14.684112 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:14.684124 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:14.684697 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:14.685258 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:14.685272 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:14.686572 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:14.686575 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:14.687524 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:14.687533 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:14.688196 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:14.688199 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:14.688381 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:14.689028 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:14.689044 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:14.693513 (+ 4469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:44:14.695484 (+ 1971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:14.695489 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:14.695535 (+ 46us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:14.695856 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:14.696192 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:14.696195 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:14.696196 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:14.696359 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:14.696893 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:44:14.696899 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:44:14.697168 (+ 269us) client_negotiation.cc:769] Sending connection context
0925 04:44:14.697338 (+ 170us) client_negotiation.cc:240] Negotiation successful
0925 04:44:14.697614 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":154,"threads_started":1}
I20260925 04:44:14.699991 1288 heartbeater.cc:347] Connected to a master server at 127.0.98.126:43625
I20260925 04:44:14.700256 1288 heartbeater.cc:464] Registering TS with master...
I20260925 04:44:14.700814 1288 heartbeater.cc:511] Master 127.0.98.126:43625 requested a full tablet report, sending...
I20260925 04:44:14.701994 963 ts_manager.cc:194] Registered new tserver with Master: e4210df3a8744694aa3920849c4bfad9 (127.0.98.66:43673)
I20260925 04:44:14.702713 963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.66@KRBTEST.COM'} at 127.0.98.66:48645
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/krb5kdc/kudu_127.0.98.67.keytab.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/krb5kdc/kudu_127.0.98.67.keytab.
I20260925 04:44:14.710011 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:0
--local_ip_for_outbound_sockets=127.0.98.67
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:43625
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:45167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:44:14.824090 1293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:44:14.824380 1293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:44:14.824426 1293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:44:14.829015 1293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:44:14.829092 1293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:44:14.829154 1293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260925 04:44:14.834754 1293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:45167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:43625
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:44:14.835999 1293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:44:14.837067 1293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:44:14.846800 1298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:14.846823 1299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:14.846900 1293 server_base.cc:1034] running on GCE node
W20260925 04:44:14.846794 1301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:14.847697 1293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:44:14.848318 1293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:44:14.849515 1293 hybrid_clock.cc:648] HybridClock initialized: now 1790311454849499 us; error 29 us; skew 500 ppm
Sep 25 04:44:14 dist-test-slave-kghz krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311454, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:44:14.852473 1293 init.cc:377] Logged in from keytab as kudu/127.0.98.67@KRBTEST.COM (short username kudu)
I20260925 04:44:14.853394 1293 webserver.cc:460] Webserver started at http://127.0.98.67:38701/ using document root <none> and password file <none>
I20260925 04:44:14.853950 1293 fs_manager.cc:359] Metadata directory not provided
I20260925 04:44:14.854012 1293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:44:14.854177 1293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:44:14.856251 1293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/minicluster-data/ts-2/data/instance:
uuid: "fa9856ed288346dbb80676ddd216a22b"
format_stamp: "Formatted at 2026-09-25 04:44:14 on dist-test-slave-kghz"
server_key: "67ba424d6df558fae84fed5544e8cbdb"
server_key_iv: "0904a55ba0e423bb2c6ea78bdec27502"
server_key_version: "encryptionkey@0"
I20260925 04:44:14.856963 1293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/minicluster-data/ts-2/wal/instance:
uuid: "fa9856ed288346dbb80676ddd216a22b"
format_stamp: "Formatted at 2026-09-25 04:44:14 on dist-test-slave-kghz"
server_key: "67ba424d6df558fae84fed5544e8cbdb"
server_key_iv: "0904a55ba0e423bb2c6ea78bdec27502"
server_key_version: "encryptionkey@0"
I20260925 04:44:14.861182 1293 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 04:44:14.863790 1308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:14.865036 1293 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 04:44:14.865173 1293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/minicluster-data/ts-2/wal
uuid: "fa9856ed288346dbb80676ddd216a22b"
format_stamp: "Formatted at 2026-09-25 04:44:14 on dist-test-slave-kghz"
server_key: "67ba424d6df558fae84fed5544e8cbdb"
server_key_iv: "0904a55ba0e423bb2c6ea78bdec27502"
server_key_version: "encryptionkey@0"
I20260925 04:44:14.865293 1293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:44:14.877723 1293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:44:14.881274 1293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:44:14.881498 1293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:44:14.882108 1293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:44:14.883018 1293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:44:14.883088 1293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:14.883160 1293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:44:14.883208 1293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:14.892303 1293 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:34199
I20260925 04:44:14.892323 1421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:34199 every 8 connection(s)
I20260925 04:44:14.893253 1293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
I20260925 04:44:14.896790 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 1293
I20260925 04:44:14.896948 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790311449873301-393-0/minicluster-data/ts-2/wal/instance
I20260925 04:44:14.897289 393 external_mini_cluster.cc:1428] Setting key 4d90686747df72d0c265c77f6ec2e1f1
Sep 25 04:44:14 dist-test-slave-kghz krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311454, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:44:14.909098 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:14.895202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:56735 (local address 127.0.98.126:43625)
0925 04:44:14.895342 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:14.895347 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:14.896195 (+ 848us) server_negotiation.cc:407] Connection header received
0925 04:44:14.898171 (+ 1976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:14.898174 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:14.898231 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:14.898391 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:14.900153 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:14.900788 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:14.902035 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:14.902448 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:14.902551 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:14.905357 (+ 2806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:14.905360 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:14.905362 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:14.905389 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:14.907007 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:14.907571 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:14.907574 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:14.907576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:14.907677 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:14.908016 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:14.908019 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:14.908020 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:14.908251 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:14.908325 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:14.908847 (+ 522us) server_negotiation.cc:299] Negotiation successful
0925 04:44:14.908970 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 04:44:14.910038 1424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:14.895559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:43625 (local address 127.0.98.67:56735)
0925 04:44:14.896028 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:14.896070 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:14.897872 (+ 1802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:14.898763 (+ 891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:14.898771 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:14.899275 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:14.899936 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:14.899953 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:14.900956 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:14.900959 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:14.901899 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:14.901909 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:14.902628 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:14.902631 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:14.902794 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:14.903429 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:14.903456 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:14.905159 (+ 1703us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:44:14.907119 (+ 1960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:14.907124 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:14.907138 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:14.907412 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:14.907777 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:14.907779 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:14.907781 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:14.907936 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:14.908346 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:44:14.908351 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:44:14.908605 (+ 254us) client_negotiation.cc:769] Sending connection context
0925 04:44:14.908766 (+ 161us) client_negotiation.cc:240] Negotiation successful
0925 04:44:14.909054 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":124,"threads_started":1}
I20260925 04:44:14.911376 1422 heartbeater.cc:347] Connected to a master server at 127.0.98.126:43625
I20260925 04:44:14.911615 1422 heartbeater.cc:464] Registering TS with master...
I20260925 04:44:14.912123 1422 heartbeater.cc:511] Master 127.0.98.126:43625 requested a full tablet report, sending...
I20260925 04:44:14.913158 963 ts_manager.cc:194] Registered new tserver with Master: fa9856ed288346dbb80676ddd216a22b (127.0.98.67:34199)
I20260925 04:44:14.913717 963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.67@KRBTEST.COM'} at 127.0.98.67:56735
I20260925 04:44:14.923977 393 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 04:44:14.935870 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:14.928100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50768 (local address 127.0.98.126:43625)
0925 04:44:14.928248 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:14.928253 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:14.928381 (+ 128us) server_negotiation.cc:407] Connection header received
0925 04:44:14.928479 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:14.928482 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:14.928525 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:14.928619 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:14.929293 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:14.929828 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:14.930793 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:14.931019 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:14.931118 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:14.932556 (+ 1438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:14.932559 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:14.932561 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:14.932586 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:14.934097 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:14.934604 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:14.934606 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:14.934608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:14.934693 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:14.934930 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:14.934933 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:14.934934 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:14.935151 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:14.935259 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:14.935504 (+ 245us) server_negotiation.cc:299] Negotiation successful
0925 04:44:14.935608 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 04:44:14.941402 963 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50768:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:44:14.944540 963 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:44:14.961326 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:14.955352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57806 (local address 127.0.98.65:42257)
0925 04:44:14.955630 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:14.955644 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:14.955664 (+ 20us) server_negotiation.cc:407] Connection header received
0925 04:44:14.955735 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:14.955741 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:14.955931 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:14.956120 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:44:14.956451 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:14.957395 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:14.959349 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:14.960215 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:14.960356 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:14.960695 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:14.961020 (+ 325us) server_negotiation.cc:299] Negotiation successful
0925 04:44:14.961139 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":71,"threads_started":1}
I20260925 04:44:14.961319 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:14.955023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:42257 (local address 127.0.0.1:57806)
0925 04:44:14.955310 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:14.955322 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:14.955414 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:14.956119 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:14.956121 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:14.956140 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:44:14.956296 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:14.956303 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:14.957534 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:14.957540 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:14.959195 (+ 1655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:14.959206 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:14.960346 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:14.960349 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:14.960692 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:14.960824 (+ 132us) client_negotiation.cc:769] Sending connection context
0925 04:44:14.960958 (+ 134us) client_negotiation.cc:240] Negotiation successful
0925 04:44:14.961160 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"mutex_wait_us":75,"thread_start_us":90,"threads_started":1}
I20260925 04:44:14.961428 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:14.954405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39716 (local address 127.0.98.66:43673)
0925 04:44:14.955164 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:14.955176 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:14.955876 (+ 700us) server_negotiation.cc:407] Connection header received
0925 04:44:14.956094 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:14.956101 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:14.956307 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:14.956471 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:44:14.956812 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:14.958042 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:14.959649 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:14.960476 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:14.960605 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:14.960756 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:14.961160 (+ 404us) server_negotiation.cc:299] Negotiation successful
0925 04:44:14.961264 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":107,"threads_started":1}
I20260925 04:44:14.961318 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:14.954200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:43673 (local address 127.0.0.1:39716)
0925 04:44:14.955786 (+ 1586us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:14.955801 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:14.955943 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:14.956601 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:14.956604 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:14.956622 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:44:14.956697 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:14.956703 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:14.958174 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:14.958178 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:14.959515 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:14.959526 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:14.960823 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:14.960858 (+ 35us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:14.960915 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:14.960964 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 04:44:14.961077 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 04:44:14.961148 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":176,"threads_started":1}
I20260925 04:44:14.962037 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:14.954270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:34199 (local address 127.0.0.1:44186)
0925 04:44:14.955075 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:14.955130 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:14.955266 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:14.956076 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:14.956082 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:14.956106 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:44:14.956429 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:14.956447 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:14.958010 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:14.958014 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:14.959217 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:14.959223 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:14.960614 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:14.960618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:14.960710 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:14.960842 (+ 132us) client_negotiation.cc:769] Sending connection context
0925 04:44:14.960958 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 04:44:14.961873 (+ 915us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"mutex_wait_us":230,"thread_start_us":85,"threads_started":1}
I20260925 04:44:14.962332 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:14.954714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44186 (local address 127.0.98.67:34199)
0925 04:44:14.955374 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:14.955385 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:14.955409 (+ 24us) server_negotiation.cc:407] Connection header received
0925 04:44:14.955490 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:14.955496 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:14.955736 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:14.955885 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:44:14.956590 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:14.957847 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:14.959404 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:14.960251 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:14.960370 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:14.960537 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:14.961404 (+ 867us) server_negotiation.cc:299] Negotiation successful
0925 04:44:14.962127 (+ 723us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":115,"threads_started":1}
I20260925 04:44:14.964939 1222 tablet_service.cc:1467] Processing CreateTablet for tablet f52417dcf1794377b3d4fe1cb63e55ec (DEFAULT_TABLE table=test-table [id=aa91cfb63e6949c4ad6a19faa4839ada]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:44:14.964943 1356 tablet_service.cc:1467] Processing CreateTablet for tablet f52417dcf1794377b3d4fe1cb63e55ec (DEFAULT_TABLE table=test-table [id=aa91cfb63e6949c4ad6a19faa4839ada]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:44:14.964943 1088 tablet_service.cc:1467] Processing CreateTablet for tablet f52417dcf1794377b3d4fe1cb63e55ec (DEFAULT_TABLE table=test-table [id=aa91cfb63e6949c4ad6a19faa4839ada]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:44:14.965977 1088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f52417dcf1794377b3d4fe1cb63e55ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:44:14.965977 1356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f52417dcf1794377b3d4fe1cb63e55ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:44:14.966007 1222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f52417dcf1794377b3d4fe1cb63e55ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:44:14.972724 1439 tablet_bootstrap.cc:492] T f52417dcf1794377b3d4fe1cb63e55ec P e4210df3a8744694aa3920849c4bfad9: Bootstrap starting.
I20260925 04:44:14.974382 1440 tablet_bootstrap.cc:492] T f52417dcf1794377b3d4fe1cb63e55ec P 9292326bc9e844d8a0a50c159b30fdcb: Bootstrap starting.
I20260925 04:44:14.974798 1441 tablet_bootstrap.cc:492] T f52417dcf1794377b3d4fe1cb63e55ec P fa9856ed288346dbb80676ddd216a22b: Bootstrap starting.
I20260925 04:44:14.976472 1439 tablet_bootstrap.cc:654] T f52417dcf1794377b3d4fe1cb63e55ec P e4210df3a8744694aa3920849c4bfad9: Neither blocks nor log segments found. Creating new log.
I20260925 04:44:14.976670 1440 tablet_bootstrap.cc:654] T f52417dcf1794377b3d4fe1cb63e55ec P 9292326bc9e844d8a0a50c159b30fdcb: Neither blocks nor log segments found. Creating new log.
I20260925 04:44:14.976698 1441 tablet_bootstrap.cc:654] T f52417dcf1794377b3d4fe1cb63e55ec P fa9856ed288346dbb80676ddd216a22b: Neither blocks nor log segments found. Creating new log.
I20260925 04:44:14.977353 1439 log.cc:824] T f52417dcf1794377b3d4fe1cb63e55ec P e4210df3a8744694aa3920849c4bfad9: Log is configured to *not* fsync() on all Append() calls
I20260925 04:44:14.978049 1440 log.cc:824] T f52417dcf1794377b3d4fe1cb63e55ec P 9292326bc9e844d8a0a50c159b30fdcb: Log is configured to *not* fsync() on all Append() calls
I20260925 04:44:14.978049 1441 log.cc:824] T f52417dcf1794377b3d4fe1cb63e55ec P fa9856ed288346dbb80676ddd216a22b: Log is configured to *not* fsync() on all Append() calls
I20260925 04:44:14.979912 1439 tablet_bootstrap.cc:492] T f52417dcf1794377b3d4fe1cb63e55ec P e4210df3a8744694aa3920849c4bfad9: No bootstrap required, opened a new log
I20260925 04:44:14.980125 1439 ts_tablet_manager.cc:1397] T f52417dcf1794377b3d4fe1cb63e55ec P e4210df3a8744694aa3920849c4bfad9: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.001s
I20260925 04:44:14.980223 1440 tablet_bootstrap.cc:492] T f52417dcf1794377b3d4fe1cb63e55ec P 9292326bc9e844d8a0a50c159b30fdcb: No bootstrap required, opened a new log
I20260925 04:44:14.980223 1441 tablet_bootstrap.cc:492] T f52417dcf1794377b3d4fe1cb63e55ec P fa9856ed288346dbb80676ddd216a22b: No bootstrap required, opened a new log
I20260925 04:44:14.980406 1440 ts_tablet_manager.cc:1397] T f52417dcf1794377b3d4fe1cb63e55ec P 9292326bc9e844d8a0a50c159b30fdcb: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 04:44:14.980406 1441 ts_tablet_manager.cc:1397] T f52417dcf1794377b3d4fe1cb63e55ec P fa9856ed288346dbb80676ddd216a22b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 04:44:14.983247 1439 raft_consensus.cc:348] T f52417dcf1794377b3d4fe1cb63e55ec P e4210df3a8744694aa3920849c4bfad9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4210df3a8744694aa3920849c4bfad9" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43673 } } peers { permanent_uuid: "fa9856ed288346dbb80676ddd216a22b" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34199 } } peers { permanent_uuid: "9292326bc9e844d8a0a50c159b30fdcb" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 42257 } }
I20260925 04:44:14.983533 1439 raft_consensus.cc:374] T f52417dcf1794377b3d4fe1cb63e55ec P e4210df3a8744694aa3920849c4bfad9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:44:14.983598 1439 raft_consensus.cc:729] T f52417dcf1794377b3d4fe1cb63e55ec P e4210df3a8744694aa3920849c4bfad9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4210df3a8744694aa3920849c4bfad9, State: Initialized, Role: FOLLOWER
I20260925 04:44:14.984097 1439 consensus_queue.cc:260] T f52417dcf1794377b3d4fe1cb63e55ec P e4210df3a8744694aa3920849c4bfad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4210df3a8744694aa3920849c4bfad9" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43673 } } peers { permanent_uuid: "fa9856ed288346dbb80676ddd216a22b" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34199 } } peers { permanent_uuid: "9292326bc9e844d8a0a50c159b30fdcb" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 42257 } }
I20260925 04:44:14.984380 1441 raft_consensus.cc:348] T f52417dcf1794377b3d4fe1cb63e55ec P fa9856ed288346dbb80676ddd216a22b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4210df3a8744694aa3920849c4bfad9" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43673 } } peers { permanent_uuid: "fa9856ed288346dbb80676ddd216a22b" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34199 } } peers { permanent_uuid: "9292326bc9e844d8a0a50c159b30fdcb" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 42257 } }
I20260925 04:44:14.984380 1440 raft_consensus.cc:348] T f52417dcf1794377b3d4fe1cb63e55ec P 9292326bc9e844d8a0a50c159b30fdcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4210df3a8744694aa3920849c4bfad9" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43673 } } peers { permanent_uuid: "fa9856ed288346dbb80676ddd216a22b" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34199 } } peers { permanent_uuid: "9292326bc9e844d8a0a50c159b30fdcb" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 42257 } }
I20260925 04:44:14.984654 1440 raft_consensus.cc:374] T f52417dcf1794377b3d4fe1cb63e55ec P 9292326bc9e844d8a0a50c159b30fdcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:44:14.984655 1441 raft_consensus.cc:374] T f52417dcf1794377b3d4fe1cb63e55ec P fa9856ed288346dbb80676ddd216a22b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:44:14.984711 1441 raft_consensus.cc:729] T f52417dcf1794377b3d4fe1cb63e55ec P fa9856ed288346dbb80676ddd216a22b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa9856ed288346dbb80676ddd216a22b, State: Initialized, Role: FOLLOWER
I20260925 04:44:14.984711 1440 raft_consensus.cc:729] T f52417dcf1794377b3d4fe1cb63e55ec P 9292326bc9e844d8a0a50c159b30fdcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9292326bc9e844d8a0a50c159b30fdcb, State: Initialized, Role: FOLLOWER
I20260925 04:44:14.985150 1288 heartbeater.cc:503] Master 127.0.98.126:43625 was elected leader, sending a full tablet report...
I20260925 04:44:14.985247 1439 ts_tablet_manager.cc:1428] T f52417dcf1794377b3d4fe1cb63e55ec P e4210df3a8744694aa3920849c4bfad9: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20260925 04:44:14.985213 1440 consensus_queue.cc:260] T f52417dcf1794377b3d4fe1cb63e55ec P 9292326bc9e844d8a0a50c159b30fdcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4210df3a8744694aa3920849c4bfad9" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43673 } } peers { permanent_uuid: "fa9856ed288346dbb80676ddd216a22b" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34199 } } peers { permanent_uuid: "9292326bc9e844d8a0a50c159b30fdcb" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 42257 } }
I20260925 04:44:14.985889 1154 heartbeater.cc:503] Master 127.0.98.126:43625 was elected leader, sending a full tablet report...
I20260925 04:44:14.986033 1441 consensus_queue.cc:260] T f52417dcf1794377b3d4fe1cb63e55ec P fa9856ed288346dbb80676ddd216a22b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4210df3a8744694aa3920849c4bfad9" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43673 } } peers { permanent_uuid: "fa9856ed288346dbb80676ddd216a22b" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34199 } } peers { permanent_uuid: "9292326bc9e844d8a0a50c159b30fdcb" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 42257 } }
I20260925 04:44:14.986174 1440 ts_tablet_manager.cc:1428] T f52417dcf1794377b3d4fe1cb63e55ec P 9292326bc9e844d8a0a50c159b30fdcb: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 04:44:14.987180 1441 ts_tablet_manager.cc:1428] T f52417dcf1794377b3d4fe1cb63e55ec P fa9856ed288346dbb80676ddd216a22b: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260925 04:44:14.987668 1422 heartbeater.cc:503] Master 127.0.98.126:43625 was elected leader, sending a full tablet report...
I20260925 04:44:15.033423 1445 raft_consensus.cc:482] T f52417dcf1794377b3d4fe1cb63e55ec P e4210df3a8744694aa3920849c4bfad9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:44:15.033676 1445 raft_consensus.cc:504] T f52417dcf1794377b3d4fe1cb63e55ec P e4210df3a8744694aa3920849c4bfad9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4210df3a8744694aa3920849c4bfad9" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43673 } } peers { permanent_uuid: "fa9856ed288346dbb80676ddd216a22b" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34199 } } peers { permanent_uuid: "9292326bc9e844d8a0a50c159b30fdcb" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 42257 } }
I20260925 04:44:15.034919 1445 leader_election.cc:288] T f52417dcf1794377b3d4fe1cb63e55ec P e4210df3a8744694aa3920849c4bfad9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa9856ed288346dbb80676ddd216a22b (127.0.98.67:34199), 9292326bc9e844d8a0a50c159b30fdcb (127.0.98.65:42257)
I20260925 04:44:15.039438 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:15.035348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:58387 (local address 127.0.98.67:34199)
0925 04:44:15.035484 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:15.035488 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:15.035504 (+ 16us) server_negotiation.cc:407] Connection header received
0925 04:44:15.035682 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:15.035685 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:15.035742 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:15.035846 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:44:15.036145 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:15.036661 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:15.038228 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:15.038735 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:15.038866 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:15.038929 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:15.039144 (+ 215us) server_negotiation.cc:299] Negotiation successful
0925 04:44:15.039290 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 04:44:15.039439 1290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:15.035265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:34199 (local address 127.0.98.66:58387)
0925 04:44:15.035405 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:15.035424 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:15.035532 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:15.035860 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:15.035863 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:15.035883 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:44:15.036021 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:15.036027 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:15.036780 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:15.036787 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:15.038099 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:15.038108 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:15.038875 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:15.038878 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:15.038984 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:15.039005 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 04:44:15.039121 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 04:44:15.039281 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":29}
I20260925 04:44:15.039777 1448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:15.035591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:42257 (local address 127.0.98.66:56313)
0925 04:44:15.035989 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:15.036000 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:15.036114 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:15.036527 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:15.036530 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:15.036594 (+ 64us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:44:15.036768 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:15.036776 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:15.037770 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:15.037774 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:15.038656 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:15.038665 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:15.039310 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:15.039313 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:15.039411 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:15.039434 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 04:44:15.039528 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 04:44:15.039625 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"mutex_wait_us":1,"thread_start_us":113,"threads_started":1}
I20260925 04:44:15.039902 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:15.035773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:56313 (local address 127.0.98.65:42257)
0925 04:44:15.035903 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:15.035908 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:15.036078 (+ 170us) server_negotiation.cc:407] Connection header received
0925 04:44:15.036225 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:15.036228 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:15.036386 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:15.036517 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:44:15.036886 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:15.037651 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:15.038788 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:15.039190 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:15.039309 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:15.039369 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:15.039693 (+ 324us) server_negotiation.cc:299] Negotiation successful
0925 04:44:15.039772 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 04:44:15.040264 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f52417dcf1794377b3d4fe1cb63e55ec" candidate_uuid: "e4210df3a8744694aa3920849c4bfad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa9856ed288346dbb80676ddd216a22b" is_pre_election: true
I20260925 04:44:15.040503 1108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f52417dcf1794377b3d4fe1cb63e55ec" candidate_uuid: "e4210df3a8744694aa3920849c4bfad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9292326bc9e844d8a0a50c159b30fdcb" is_pre_election: true
I20260925 04:44:15.040560 1375 raft_consensus.cc:2454] T f52417dcf1794377b3d4fe1cb63e55ec P fa9856ed288346dbb80676ddd216a22b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4210df3a8744694aa3920849c4bfad9 in term 0.
I20260925 04:44:15.040810 1108 raft_consensus.cc:2454] T f52417dcf1794377b3d4fe1cb63e55ec P 9292326bc9e844d8a0a50c159b30fdcb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4210df3a8744694aa3920849c4bfad9 in term 0.
I20260925 04:44:15.041072 1178 leader_election.cc:302] T f52417dcf1794377b3d4fe1cb63e55ec P e4210df3a8744694aa3920849c4bfad9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e4210df3a8744694aa3920849c4bfad9, fa9856ed288346dbb80676ddd216a22b; no voters:
I20260925 04:44:15.041364 1445 raft_consensus.cc:2781] T f52417dcf1794377b3d4fe1cb63e55ec P e4210df3a8744694aa3920849c4bfad9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:44:15.041463 1445 raft_consensus.cc:482] T f52417dcf1794377b3d4fe1cb63e55ec P e4210df3a8744694aa3920849c4bfad9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:44:15.041523 1445 raft_consensus.cc:3037] T f52417dcf1794377b3d4fe1cb63e55ec P e4210df3a8744694aa3920849c4bfad9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:44:15.042665 1445 raft_consensus.cc:504] T f52417dcf1794377b3d4fe1cb63e55ec P e4210df3a8744694aa3920849c4bfad9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4210df3a8744694aa3920849c4bfad9" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43673 } } peers { permanent_uuid: "fa9856ed288346dbb80676ddd216a22b" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34199 } } peers { permanent_uuid: "9292326bc9e844d8a0a50c159b30fdcb" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 42257 } }
I20260925 04:44:15.043028 1445 leader_election.cc:288] T f52417dcf1794377b3d4fe1cb63e55ec P e4210df3a8744694aa3920849c4bfad9 [CANDIDATE]: Term 1 election: Requested vote from peers fa9856ed288346dbb80676ddd216a22b (127.0.98.67:34199), 9292326bc9e844d8a0a50c159b30fdcb (127.0.98.65:42257)
I20260925 04:44:15.043448 1108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f52417dcf1794377b3d4fe1cb63e55ec" candidate_uuid: "e4210df3a8744694aa3920849c4bfad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9292326bc9e844d8a0a50c159b30fdcb"
I20260925 04:44:15.043445 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f52417dcf1794377b3d4fe1cb63e55ec" candidate_uuid: "e4210df3a8744694aa3920849c4bfad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa9856ed288346dbb80676ddd216a22b"
I20260925 04:44:15.043584 1108 raft_consensus.cc:3037] T f52417dcf1794377b3d4fe1cb63e55ec P 9292326bc9e844d8a0a50c159b30fdcb [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:44:15.043598 1375 raft_consensus.cc:3037] T f52417dcf1794377b3d4fe1cb63e55ec P fa9856ed288346dbb80676ddd216a22b [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:44:15.044625 1108 raft_consensus.cc:2454] T f52417dcf1794377b3d4fe1cb63e55ec P 9292326bc9e844d8a0a50c159b30fdcb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4210df3a8744694aa3920849c4bfad9 in term 1.
I20260925 04:44:15.044636 1375 raft_consensus.cc:2454] T f52417dcf1794377b3d4fe1cb63e55ec P fa9856ed288346dbb80676ddd216a22b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4210df3a8744694aa3920849c4bfad9 in term 1.
I20260925 04:44:15.044936 1178 leader_election.cc:302] T f52417dcf1794377b3d4fe1cb63e55ec P e4210df3a8744694aa3920849c4bfad9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9292326bc9e844d8a0a50c159b30fdcb, e4210df3a8744694aa3920849c4bfad9; no voters:
I20260925 04:44:15.045143 1445 raft_consensus.cc:2781] T f52417dcf1794377b3d4fe1cb63e55ec P e4210df3a8744694aa3920849c4bfad9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:44:15.045430 1445 raft_consensus.cc:686] T f52417dcf1794377b3d4fe1cb63e55ec P e4210df3a8744694aa3920849c4bfad9 [term 1 LEADER]: Becoming Leader. State: Replica: e4210df3a8744694aa3920849c4bfad9, State: Running, Role: LEADER
I20260925 04:44:15.045800 1445 consensus_queue.cc:237] T f52417dcf1794377b3d4fe1cb63e55ec P e4210df3a8744694aa3920849c4bfad9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4210df3a8744694aa3920849c4bfad9" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43673 } } peers { permanent_uuid: "fa9856ed288346dbb80676ddd216a22b" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34199 } } peers { permanent_uuid: "9292326bc9e844d8a0a50c159b30fdcb" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 42257 } }
I20260925 04:44:15.049237 962 catalog_manager.cc:5495] T f52417dcf1794377b3d4fe1cb63e55ec P e4210df3a8744694aa3920849c4bfad9 reported cstate change: term changed from 0 to 1, leader changed from <none> to e4210df3a8744694aa3920849c4bfad9 (127.0.98.66). New cstate: current_term: 1 leader_uuid: "e4210df3a8744694aa3920849c4bfad9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4210df3a8744694aa3920849c4bfad9" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa9856ed288346dbb80676ddd216a22b" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9292326bc9e844d8a0a50c159b30fdcb" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 42257 } health_report { overall_health: UNKNOWN } } }
Sep 25 04:44:15 dist-test-slave-kghz krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.65@KRBTEST.COM
I20260925 04:44:15.068919 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:15.058943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57820 (local address 127.0.98.65:42257)
0925 04:44:15.059074 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:15.059077 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:15.059295 (+ 218us) server_negotiation.cc:407] Connection header received
0925 04:44:15.059373 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:15.059376 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:15.059422 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:15.059495 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:15.060357 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:15.060934 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:15.061880 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:15.062103 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:15.062311 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:15.064943 (+ 2632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:15.064957 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:15.064972 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:15.065000 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:15.066956 (+ 1956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:15.067381 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:15.067387 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:15.067388 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:15.067475 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:15.067853 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:15.067856 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:15.067857 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:15.068132 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:15.068344 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:15.068650 (+ 306us) server_negotiation.cc:299] Negotiation successful
0925 04:44:15.068768 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
Sep 25 04:44:15 dist-test-slave-kghz krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311455, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 04:44:15 dist-test-slave-kghz krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311455, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.65@KRBTEST.COM
I20260925 04:44:15.097291 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:15.088425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57828 (local address 127.0.98.65:42257)
0925 04:44:15.088588 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:15.088592 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:15.088622 (+ 30us) server_negotiation.cc:407] Connection header received
0925 04:44:15.088770 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:15.088774 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:15.088818 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:15.088897 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:15.089637 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:15.090164 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:15.091092 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:15.091345 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:15.091428 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:15.093977 (+ 2549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:15.093981 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:15.093983 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:15.094008 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:15.095416 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:15.095890 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:15.095893 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:15.095895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:15.095980 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:15.096279 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:15.096282 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:15.096283 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:15.096461 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:15.096570 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:15.096901 (+ 331us) server_negotiation.cc:299] Negotiation successful
0925 04:44:15.097015 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
Sep 25 04:44:15 dist-test-slave-kghz krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311455, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 04:44:15 dist-test-slave-kghz krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311455, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.65@KRBTEST.COM
I20260925 04:44:15.124294 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:15.115049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57844 (local address 127.0.98.65:42257)
0925 04:44:15.115219 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:15.115225 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:15.115401 (+ 176us) server_negotiation.cc:407] Connection header received
0925 04:44:15.115640 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:15.115644 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:15.115703 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:15.115797 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:15.116565 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:15.117113 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:15.117972 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:15.118223 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:15.118366 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:15.120970 (+ 2604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:15.120974 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:15.120976 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:15.121010 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:15.122630 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:15.123061 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:15.123064 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:15.123066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:15.123155 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:15.123430 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:15.123435 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:15.123436 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:15.123610 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:15.123748 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:15.123989 (+ 241us) server_negotiation.cc:299] Negotiation successful
0925 04:44:15.124112 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260925 04:44:15.125797 1068 tablet_service.cc:2908] Rejecting scan request for tablet f52417dcf1794377b3d4fe1cb63e55ec: Uninitialized: clean time has not yet been initialized
I20260925 04:44:15.128060 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 1025
I20260925 04:44:15.134632 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 1159
I20260925 04:44:15.140883 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 1293
I20260925 04:44:15.147580 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 931
2026-09-25T04:44:15Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2399 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:44:15 dist-test-slave-kghz krb5kdc[1463](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:44:15 dist-test-slave-kghz krb5kdc[1463](info): set up 2 sockets
Sep 25 04:44:15 dist-test-slave-kghz krb5kdc[1463](info): commencing operation
krb5kdc: starting...
W20260925 04:44:16.202008 393 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/krb5kdc/test-user.keytab.
Sep 25 04:44:16 dist-test-slave-kghz krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:44:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:44:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260925 04:44:16.330593 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:36341
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:41007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:36341
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:44:16.441674 1476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:44:16.441982 1476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:44:16.442024 1476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:44:16.446177 1476 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:44:16.446264 1476 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 04:44:16.446326 1476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:44:16.446367 1476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:44:16.446400 1476 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:44:16.446439 1476 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:44:16.451996 1476 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:41007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:36341
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:36341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:44:16.453118 1476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:44:16.454355 1476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:44:16.462875 1482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:16.462903 1484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:16.463142 1476 server_base.cc:1034] running on GCE node
W20260925 04:44:16.462882 1481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:16.463855 1476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:44:16.465034 1476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:44:16.466229 1476 hybrid_clock.cc:648] HybridClock initialized: now 1790311456466190 us; error 53 us; skew 500 ppm
Sep 25 04:44:16 dist-test-slave-kghz krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311456, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:44:16.474512 1476 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260925 04:44:16.475522 1476 webserver.cc:460] Webserver started at http://127.0.98.126:44535/ using document root <none> and password file <none>
I20260925 04:44:16.476130 1476 fs_manager.cc:359] Metadata directory not provided
I20260925 04:44:16.476224 1476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:44:16.476464 1476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:44:16.478726 1476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/minicluster-data/master-0/data/instance:
uuid: "6f7507ec830a4633868cda169231b809"
format_stamp: "Formatted at 2026-09-25 04:44:16 on dist-test-slave-kghz"
server_key: "499678c07cf651c1e163b581e3e30ddf"
server_key_iv: "568a33c12253d5837d30f336d4a1a43d"
server_key_version: "encryptionkey@0"
I20260925 04:44:16.479315 1476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/minicluster-data/master-0/wal/instance:
uuid: "6f7507ec830a4633868cda169231b809"
format_stamp: "Formatted at 2026-09-25 04:44:16 on dist-test-slave-kghz"
server_key: "499678c07cf651c1e163b581e3e30ddf"
server_key_iv: "568a33c12253d5837d30f336d4a1a43d"
server_key_version: "encryptionkey@0"
I20260925 04:44:16.483206 1476 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 04:44:16.485674 1491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:16.486948 1476 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 04:44:16.487084 1476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/minicluster-data/master-0/wal
uuid: "6f7507ec830a4633868cda169231b809"
format_stamp: "Formatted at 2026-09-25 04:44:16 on dist-test-slave-kghz"
server_key: "499678c07cf651c1e163b581e3e30ddf"
server_key_iv: "568a33c12253d5837d30f336d4a1a43d"
server_key_version: "encryptionkey@0"
I20260925 04:44:16.487214 1476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:44:16.517818 1476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:44:16.521502 1476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:44:16.521727 1476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:44:16.530556 1476 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:36341
I20260925 04:44:16.530597 1553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:36341 every 8 connection(s)
I20260925 04:44:16.531569 1476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
I20260925 04:44:16.534552 1554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:44:16.537523 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 1476
I20260925 04:44:16.537652 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/minicluster-data/master-0/wal/instance
I20260925 04:44:16.537896 393 external_mini_cluster.cc:1428] Setting key 63bc52ea56dc7bebcb499fabc9c927f5
I20260925 04:44:16.540804 1554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f7507ec830a4633868cda169231b809: Bootstrap starting.
I20260925 04:44:16.543087 1554 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f7507ec830a4633868cda169231b809: Neither blocks nor log segments found. Creating new log.
I20260925 04:44:16.543846 1554 log.cc:824] T 00000000000000000000000000000000 P 6f7507ec830a4633868cda169231b809: Log is configured to *not* fsync() on all Append() calls
I20260925 04:44:16.545814 1554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f7507ec830a4633868cda169231b809: No bootstrap required, opened a new log
Sep 25 04:44:16 dist-test-slave-kghz krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:44:16.549335 1554 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f7507ec830a4633868cda169231b809 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f7507ec830a4633868cda169231b809" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 36341 } }
I20260925 04:44:16.549556 1554 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f7507ec830a4633868cda169231b809 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:44:16.549626 1554 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f7507ec830a4633868cda169231b809 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f7507ec830a4633868cda169231b809, State: Initialized, Role: FOLLOWER
I20260925 04:44:16.550088 1554 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f7507ec830a4633868cda169231b809 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f7507ec830a4633868cda169231b809" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 36341 } }
I20260925 04:44:16.550215 1554 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f7507ec830a4633868cda169231b809 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:44:16.550369 1554 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f7507ec830a4633868cda169231b809 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:44:16.550500 1554 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f7507ec830a4633868cda169231b809 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:44:16.551781 1554 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f7507ec830a4633868cda169231b809 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f7507ec830a4633868cda169231b809" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 36341 } }
I20260925 04:44:16.552189 1554 leader_election.cc:302] T 00000000000000000000000000000000 P 6f7507ec830a4633868cda169231b809 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f7507ec830a4633868cda169231b809; no voters:
I20260925 04:44:16.552531 1554 leader_election.cc:288] T 00000000000000000000000000000000 P 6f7507ec830a4633868cda169231b809 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:44:16.552695 1559 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f7507ec830a4633868cda169231b809 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:44:16.552973 1559 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f7507ec830a4633868cda169231b809 [term 1 LEADER]: Becoming Leader. State: Replica: 6f7507ec830a4633868cda169231b809, State: Running, Role: LEADER
I20260925 04:44:16.553218 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:16.539496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48954 (local address 127.0.98.126:36341)
0925 04:44:16.539976 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:16.539991 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:16.540030 (+ 39us) server_negotiation.cc:407] Connection header received
0925 04:44:16.540689 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:16.540706 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:16.541123 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:16.541482 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:16.542302 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:16.543363 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:16.544289 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:16.544775 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:16.544956 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:16.547757 (+ 2801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:16.547762 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:16.547776 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:16.547803 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:16.549597 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:16.550127 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:16.550133 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:16.550145 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:16.550368 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:16.550673 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:16.550677 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:16.550679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:16.551262 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:16.551482 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:16.551778 (+ 296us) server_negotiation.cc:299] Negotiation successful
0925 04:44:16.552098 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":152,"threads_started":1}
I20260925 04:44:16.553447 1559 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f7507ec830a4633868cda169231b809 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f7507ec830a4633868cda169231b809" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 36341 } }
I20260925 04:44:16.553864 1554 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f7507ec830a4633868cda169231b809 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:44:16.554774 1559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f7507ec830a4633868cda169231b809 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f7507ec830a4633868cda169231b809. Latest consensus state: current_term: 1 leader_uuid: "6f7507ec830a4633868cda169231b809" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f7507ec830a4633868cda169231b809" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 36341 } } }
I20260925 04:44:16.554991 1559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f7507ec830a4633868cda169231b809 [sys.catalog]: This master's current role is: LEADER
I20260925 04:44:16.555693 1568 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:44:16.556461 1560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f7507ec830a4633868cda169231b809 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f7507ec830a4633868cda169231b809" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f7507ec830a4633868cda169231b809" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 36341 } } }
I20260925 04:44:16.556564 1560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f7507ec830a4633868cda169231b809 [sys.catalog]: This master's current role is: LEADER
I20260925 04:44:16.559181 1568 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:44:16.564971 1568 catalog_manager.cc:1347] Generated new cluster ID: 0a8f035baeac49a883b0361a12a60452
I20260925 04:44:16.565090 1568 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:44:16.591158 1568 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:44:16.592710 1568 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:44:16.601864 1568 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f7507ec830a4633868cda169231b809: Generated new TSK 0
I20260925 04:44:16.602790 1568 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/krb5kdc/kudu_127.0.98.65.keytab.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/krb5kdc/kudu_127.0.98.65.keytab.
I20260925 04:44:16.638700 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:0
--local_ip_for_outbound_sockets=127.0.98.65
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:36341
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:41007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:44:16.754390 1580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:44:16.754755 1580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:44:16.754833 1580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:44:16.759162 1580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:44:16.759229 1580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:44:16.759270 1580 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 04:44:16.759338 1580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260925 04:44:16.764739 1580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:41007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.126:36341
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:44:16.765863 1580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:44:16.766990 1580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:44:16.776695 1588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:16.776695 1586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:16.776770 1585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:16.776971 1580 server_base.cc:1034] running on GCE node
I20260925 04:44:16.777508 1580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:44:16.778219 1580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:44:16.779465 1580 hybrid_clock.cc:648] HybridClock initialized: now 1790311456779430 us; error 48 us; skew 500 ppm
Sep 25 04:44:16 dist-test-slave-kghz krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311456, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:44:16.782392 1580 init.cc:377] Logged in from keytab as kudu/127.0.98.65@KRBTEST.COM (short username kudu)
I20260925 04:44:16.783311 1580 webserver.cc:460] Webserver started at http://127.0.98.65:43113/ using document root <none> and password file <none>
I20260925 04:44:16.783877 1580 fs_manager.cc:359] Metadata directory not provided
I20260925 04:44:16.783972 1580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:44:16.784202 1580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:44:16.786365 1580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/minicluster-data/ts-0/data/instance:
uuid: "2a5a1853f59b4b63a2f6cd0c618f97f1"
format_stamp: "Formatted at 2026-09-25 04:44:16 on dist-test-slave-kghz"
server_key: "e587784684e0f60e70cc2d4c70f9ef48"
server_key_iv: "fcd2f91cd92e6d5397069655d31cb97d"
server_key_version: "encryptionkey@0"
I20260925 04:44:16.786958 1580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/minicluster-data/ts-0/wal/instance:
uuid: "2a5a1853f59b4b63a2f6cd0c618f97f1"
format_stamp: "Formatted at 2026-09-25 04:44:16 on dist-test-slave-kghz"
server_key: "e587784684e0f60e70cc2d4c70f9ef48"
server_key_iv: "fcd2f91cd92e6d5397069655d31cb97d"
server_key_version: "encryptionkey@0"
I20260925 04:44:16.790885 1580 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260925 04:44:16.793570 1595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:16.794967 1580 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:44:16.795141 1580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/minicluster-data/ts-0/wal
uuid: "2a5a1853f59b4b63a2f6cd0c618f97f1"
format_stamp: "Formatted at 2026-09-25 04:44:16 on dist-test-slave-kghz"
server_key: "e587784684e0f60e70cc2d4c70f9ef48"
server_key_iv: "fcd2f91cd92e6d5397069655d31cb97d"
server_key_version: "encryptionkey@0"
I20260925 04:44:16.795270 1580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:44:16.812098 1580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:44:16.815644 1580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:44:16.815862 1580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:44:16.817392 1580 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:44:16.817466 1580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:16.817540 1580 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:44:16.817582 1580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 04:44:16 dist-test-slave-kghz krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311456, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:44:16.828630 1580 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:43321
I20260925 04:44:16.828701 1710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:43321 every 8 connection(s)
I20260925 04:44:16.829540 1580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
I20260925 04:44:16.832396 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:16.818190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:35565 (local address 127.0.98.126:36341)
0925 04:44:16.818362 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:16.818368 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:16.819316 (+ 948us) server_negotiation.cc:407] Connection header received
0925 04:44:16.820640 (+ 1324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:16.820645 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:16.820728 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:16.820795 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:16.822785 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:16.823437 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:16.824596 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:16.824924 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:16.825040 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:16.828635 (+ 3595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:16.828642 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:16.828645 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:16.828679 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:16.830391 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:16.830933 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:16.830936 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:16.830937 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:16.831015 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:16.831396 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:16.831399 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:16.831400 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:16.831594 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:16.831678 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:16.832139 (+ 461us) server_negotiation.cc:299] Negotiation successful
0925 04:44:16.832270 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 04:44:16.833202 1604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:16.818598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:36341 (local address 127.0.98.65:35565)
0925 04:44:16.819102 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:16.819165 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:16.820347 (+ 1182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:16.821252 (+ 905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:16.821265 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:16.821769 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:16.822604 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:16.822631 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:16.823575 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:16.823578 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:16.824466 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:16.824475 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:16.825089 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:16.825093 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:16.825312 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:16.826556 (+ 1244us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:16.826573 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:16.828291 (+ 1718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:44:16.830517 (+ 2226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:16.830523 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:16.830537 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:16.830827 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:16.831098 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:16.831101 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:16.831102 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:16.831233 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:16.831687 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:44:16.831693 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:44:16.831919 (+ 226us) client_negotiation.cc:769] Sending connection context
0925 04:44:16.832068 (+ 149us) client_negotiation.cc:240] Negotiation successful
0925 04:44:16.832325 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":106,"threads_started":1}
I20260925 04:44:16.834762 1711 heartbeater.cc:347] Connected to a master server at 127.0.98.126:36341
I20260925 04:44:16.834956 1711 heartbeater.cc:464] Registering TS with master...
I20260925 04:44:16.835578 1711 heartbeater.cc:511] Master 127.0.98.126:36341 requested a full tablet report, sending...
I20260925 04:44:16.835744 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 1580
I20260925 04:44:16.835856 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/minicluster-data/ts-0/wal/instance
I20260925 04:44:16.836148 393 external_mini_cluster.cc:1428] Setting key cfad526caecadc245ae607665ad3c562
I20260925 04:44:16.837487 1507 ts_manager.cc:194] Registered new tserver with Master: 2a5a1853f59b4b63a2f6cd0c618f97f1 (127.0.98.65:43321)
I20260925 04:44:16.839061 1507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.65@KRBTEST.COM'} at 127.0.98.65:35565
I20260925 04:44:16.844990 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:16.837164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:55893 (local address 127.0.98.126:36341)
0925 04:44:16.837272 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:16.837277 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:16.837516 (+ 239us) server_negotiation.cc:407] Connection header received
0925 04:44:16.837683 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:16.837686 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:16.837737 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:16.837828 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:16.838444 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:16.839193 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:16.840196 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:16.840429 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:16.840540 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:16.841710 (+ 1170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:16.841714 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:16.841717 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:16.841744 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:16.843312 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:16.843772 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:16.843776 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:16.843777 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:16.843862 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:16.844157 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:16.844162 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:16.844164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:16.844342 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:16.844449 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:16.844713 (+ 264us) server_negotiation.cc:299] Negotiation successful
0925 04:44:16.844823 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 04:44:16.845011 1717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:16.837125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:36341 (local address 127.0.98.65:55893)
0925 04:44:16.837449 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:16.837465 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:16.837557 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:16.837868 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:16.837871 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:16.838123 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:16.838316 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:16.838327 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:16.839346 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:16.839350 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:16.840063 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:16.840080 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:16.840545 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:16.840549 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:16.840640 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:16.841199 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:16.841220 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:16.841559 (+ 339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:44:16.843446 (+ 1887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:16.843449 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:16.843452 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:16.843662 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:16.843954 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:16.843958 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:16.843960 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:16.844050 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:16.844448 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:44:16.844451 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:44:16.844550 (+ 99us) client_negotiation.cc:769] Sending connection context
0925 04:44:16.844659 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 04:44:16.844815 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":98,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/krb5kdc/kudu_127.0.98.66.keytab.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/krb5kdc/kudu_127.0.98.66.keytab.
I20260925 04:44:16.869880 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:0
--local_ip_for_outbound_sockets=127.0.98.66
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:36341
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:41007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:44:16.982565 1720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:44:16.982844 1720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:44:16.982918 1720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:44:16.987071 1720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:44:16.987134 1720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:44:16.987174 1720 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 04:44:16.987246 1720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260925 04:44:16.992627 1720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:41007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.126:36341
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:44:16.993732 1720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:44:16.994787 1720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:44:17.004702 1726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:17.004702 1728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:17.004856 1720 server_base.cc:1034] running on GCE node
W20260925 04:44:17.004702 1725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:17.005458 1720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:44:17.006124 1720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:44:17.007308 1720 hybrid_clock.cc:648] HybridClock initialized: now 1790311457007271 us; error 48 us; skew 500 ppm
Sep 25 04:44:17 dist-test-slave-kghz krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311457, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:44:17.010161 1720 init.cc:377] Logged in from keytab as kudu/127.0.98.66@KRBTEST.COM (short username kudu)
I20260925 04:44:17.011065 1720 webserver.cc:460] Webserver started at http://127.0.98.66:37985/ using document root <none> and password file <none>
I20260925 04:44:17.011628 1720 fs_manager.cc:359] Metadata directory not provided
I20260925 04:44:17.011688 1720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:44:17.011904 1720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:44:17.013944 1720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/minicluster-data/ts-1/data/instance:
uuid: "75fd7b5a529546e4b2428c240711a58b"
format_stamp: "Formatted at 2026-09-25 04:44:17 on dist-test-slave-kghz"
server_key: "39a08b699f5cce1cffb9c7d3f563ac5d"
server_key_iv: "00a1fa2059cf906c7a2768446c842171"
server_key_version: "encryptionkey@0"
I20260925 04:44:17.014581 1720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/minicluster-data/ts-1/wal/instance:
uuid: "75fd7b5a529546e4b2428c240711a58b"
format_stamp: "Formatted at 2026-09-25 04:44:17 on dist-test-slave-kghz"
server_key: "39a08b699f5cce1cffb9c7d3f563ac5d"
server_key_iv: "00a1fa2059cf906c7a2768446c842171"
server_key_version: "encryptionkey@0"
I20260925 04:44:17.018451 1720 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 04:44:17.020855 1735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:17.022145 1720 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 04:44:17.022315 1720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/minicluster-data/ts-1/wal
uuid: "75fd7b5a529546e4b2428c240711a58b"
format_stamp: "Formatted at 2026-09-25 04:44:17 on dist-test-slave-kghz"
server_key: "39a08b699f5cce1cffb9c7d3f563ac5d"
server_key_iv: "00a1fa2059cf906c7a2768446c842171"
server_key_version: "encryptionkey@0"
I20260925 04:44:17.022424 1720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:44:17.039793 1720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:44:17.043331 1720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:44:17.043558 1720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:44:17.045109 1720 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:44:17.045161 1720 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:17.045202 1720 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:44:17.045270 1720 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 04:44:17 dist-test-slave-kghz krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311457, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:44:17.056890 1720 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:37773
I20260925 04:44:17.057284 1850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:37773 every 8 connection(s)
I20260925 04:44:17.057989 1720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
I20260925 04:44:17.060195 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:17.046305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:41669 (local address 127.0.98.126:36341)
0925 04:44:17.046420 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:17.046423 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:17.047555 (+ 1132us) server_negotiation.cc:407] Connection header received
0925 04:44:17.048856 (+ 1301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:17.048860 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:17.048933 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:17.049038 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:17.050654 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:17.051550 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:17.053035 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:17.053271 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:17.053388 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:17.056440 (+ 3052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:17.056446 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:17.056449 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:17.056479 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:17.058156 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:17.058821 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:17.058825 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:17.058827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:17.058918 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:17.059220 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:17.059222 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:17.059223 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:17.059397 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:17.059474 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:17.059971 (+ 497us) server_negotiation.cc:299] Negotiation successful
0925 04:44:17.060075 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 04:44:17.061074 1744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:17.046895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:36341 (local address 127.0.98.66:41669)
0925 04:44:17.047394 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:17.047438 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:17.048583 (+ 1145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:17.049267 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:17.049275 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:17.049770 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:17.050488 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:17.050505 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:17.051700 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:17.051704 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:17.052906 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:17.052916 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:17.053436 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:17.053440 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:17.053638 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:17.054314 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:17.054339 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:17.056234 (+ 1895us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:44:17.058300 (+ 2066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:17.058308 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:17.058327 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:17.058691 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:17.058998 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:17.059000 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:17.059002 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:17.059136 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:17.059517 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:44:17.059523 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:44:17.059763 (+ 240us) client_negotiation.cc:769] Sending connection context
0925 04:44:17.059919 (+ 156us) client_negotiation.cc:240] Negotiation successful
0925 04:44:17.060203 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":141,"threads_started":1}
I20260925 04:44:17.062233 1851 heartbeater.cc:347] Connected to a master server at 127.0.98.126:36341
I20260925 04:44:17.062423 1851 heartbeater.cc:464] Registering TS with master...
I20260925 04:44:17.062925 1851 heartbeater.cc:511] Master 127.0.98.126:36341 requested a full tablet report, sending...
I20260925 04:44:17.064018 1508 ts_manager.cc:194] Registered new tserver with Master: 75fd7b5a529546e4b2428c240711a58b (127.0.98.66:37773)
I20260925 04:44:17.064739 1508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.66@KRBTEST.COM'} at 127.0.98.66:41669
I20260925 04:44:17.066402 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 1720
I20260925 04:44:17.066521 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/minicluster-data/ts-1/wal/instance
I20260925 04:44:17.066819 393 external_mini_cluster.cc:1428] Setting key 138aa143b576e436d593edf9df498677
I20260925 04:44:17.072541 1857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:17.064473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:36341 (local address 127.0.98.66:60173)
0925 04:44:17.064765 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:17.064780 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:17.064880 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:17.065150 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:17.065153 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:17.065383 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:17.065478 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:17.065484 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:17.066502 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:17.066507 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:17.067328 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:17.067337 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:17.067791 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:17.067795 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:17.067885 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:17.068389 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:17.068407 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:17.068870 (+ 463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:44:17.070869 (+ 1999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:17.070873 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:17.070875 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:17.071090 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:17.071427 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:17.071430 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:17.071431 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:17.071520 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:17.072099 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:44:17.072101 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:44:17.072186 (+ 85us) client_negotiation.cc:769] Sending connection context
0925 04:44:17.072278 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 04:44:17.072395 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":117,"threads_started":1}
I20260925 04:44:17.072748 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:17.064543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:60173 (local address 127.0.98.126:36341)
0925 04:44:17.064671 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:17.064676 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:17.064860 (+ 184us) server_negotiation.cc:407] Connection header received
0925 04:44:17.064956 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:17.064960 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:17.065033 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:17.065118 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:17.065591 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:17.066367 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:17.067455 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:17.067681 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:17.067790 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:17.069015 (+ 1225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:17.069018 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:17.069020 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:17.069042 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:17.070735 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:17.071229 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:17.071235 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:17.071240 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:17.071333 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:17.071649 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:17.071655 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:17.071659 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:17.071985 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:17.072093 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:17.072365 (+ 272us) server_negotiation.cc:299] Negotiation successful
0925 04:44:17.072501 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
WARNING: no policy specified for kudu/127.0.98.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/krb5kdc/kudu_127.0.98.67.keytab.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/krb5kdc/kudu_127.0.98.67.keytab.
I20260925 04:44:17.097325 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:0
--local_ip_for_outbound_sockets=127.0.98.67
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:36341
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:41007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:44:17.209724 1860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:44:17.209987 1860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:44:17.210033 1860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:44:17.214159 1860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:44:17.214221 1860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:44:17.214275 1860 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 04:44:17.214309 1860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260925 04:44:17.219549 1860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:41007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.126:36341
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:44:17.220609 1860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:44:17.221638 1860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:44:17.231632 1868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:17.231688 1866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:17.231695 1865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:17.231807 1860 server_base.cc:1034] running on GCE node
I20260925 04:44:17.232275 1860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:44:17.233033 1860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:44:17.234232 1860 hybrid_clock.cc:648] HybridClock initialized: now 1790311457234193 us; error 51 us; skew 500 ppm
Sep 25 04:44:17 dist-test-slave-kghz krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311457, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:44:17.237323 1860 init.cc:377] Logged in from keytab as kudu/127.0.98.67@KRBTEST.COM (short username kudu)
I20260925 04:44:17.238418 1860 webserver.cc:460] Webserver started at http://127.0.98.67:42059/ using document root <none> and password file <none>
I20260925 04:44:17.239014 1860 fs_manager.cc:359] Metadata directory not provided
I20260925 04:44:17.239107 1860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:44:17.239321 1860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:44:17.241433 1860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/minicluster-data/ts-2/data/instance:
uuid: "2f218b5c8db54a2ba8ecddc69387a36a"
format_stamp: "Formatted at 2026-09-25 04:44:17 on dist-test-slave-kghz"
server_key: "7f33c57da8b6b79ceaebcefb32f1e47a"
server_key_iv: "f0bd85632d1b932359926d8268f1e70f"
server_key_version: "encryptionkey@0"
I20260925 04:44:17.242015 1860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/minicluster-data/ts-2/wal/instance:
uuid: "2f218b5c8db54a2ba8ecddc69387a36a"
format_stamp: "Formatted at 2026-09-25 04:44:17 on dist-test-slave-kghz"
server_key: "7f33c57da8b6b79ceaebcefb32f1e47a"
server_key_iv: "f0bd85632d1b932359926d8268f1e70f"
server_key_version: "encryptionkey@0"
I20260925 04:44:17.245908 1860 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260925 04:44:17.248412 1875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:17.249709 1860 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 04:44:17.249861 1860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/minicluster-data/ts-2/wal
uuid: "2f218b5c8db54a2ba8ecddc69387a36a"
format_stamp: "Formatted at 2026-09-25 04:44:17 on dist-test-slave-kghz"
server_key: "7f33c57da8b6b79ceaebcefb32f1e47a"
server_key_iv: "f0bd85632d1b932359926d8268f1e70f"
server_key_version: "encryptionkey@0"
I20260925 04:44:17.249979 1860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:44:17.266938 1860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:44:17.270239 1860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:44:17.270505 1860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:44:17.272046 1860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:44:17.272166 1860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:17.272243 1860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:44:17.272287 1860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 04:44:17 dist-test-slave-kghz krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311457, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:44:17.283917 1860 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:46817
I20260925 04:44:17.283931 1990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:46817 every 8 connection(s)
I20260925 04:44:17.285136 1860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
I20260925 04:44:17.286475 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:17.272857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:43507 (local address 127.0.98.126:36341)
0925 04:44:17.273003 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:17.273008 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:17.274007 (+ 999us) server_negotiation.cc:407] Connection header received
0925 04:44:17.275151 (+ 1144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:17.275155 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:17.275217 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:17.275281 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:17.276837 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:17.277430 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:17.278718 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:17.279135 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:17.279281 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:17.282441 (+ 3160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:17.282445 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:17.282447 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:17.282483 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:17.284095 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:17.284639 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:17.284642 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:17.284644 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:17.284744 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:17.285175 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:17.285178 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:17.285182 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:17.285413 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:17.285522 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:17.286168 (+ 646us) server_negotiation.cc:299] Negotiation successful
0925 04:44:17.286345 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 04:44:17.287245 1884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:17.273346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:36341 (local address 127.0.98.67:43507)
0925 04:44:17.273832 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:17.273882 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:17.274886 (+ 1004us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:17.275765 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:17.275771 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:17.276105 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:17.276680 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:17.276700 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:17.277569 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:17.277573 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:17.278576 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:17.278586 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:17.279332 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:17.279340 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:17.279567 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:17.280289 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:17.280314 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:17.282210 (+ 1896us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:44:17.284256 (+ 2046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:17.284261 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:17.284273 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:17.284516 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:17.284903 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:17.284906 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:17.284908 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:17.285052 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:17.285539 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:44:17.285545 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:44:17.285855 (+ 310us) client_negotiation.cc:769] Sending connection context
0925 04:44:17.286052 (+ 197us) client_negotiation.cc:240] Negotiation successful
0925 04:44:17.286415 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":139,"threads_started":1}
I20260925 04:44:17.288492 1991 heartbeater.cc:347] Connected to a master server at 127.0.98.126:36341
I20260925 04:44:17.288650 1991 heartbeater.cc:464] Registering TS with master...
I20260925 04:44:17.289219 1991 heartbeater.cc:511] Master 127.0.98.126:36341 requested a full tablet report, sending...
I20260925 04:44:17.290468 1507 ts_manager.cc:194] Registered new tserver with Master: 2f218b5c8db54a2ba8ecddc69387a36a (127.0.98.67:46817)
I20260925 04:44:17.291247 1507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.67@KRBTEST.COM'} at 127.0.98.67:43507
I20260925 04:44:17.294025 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 1860
I20260925 04:44:17.294234 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/minicluster-data/ts-2/wal/instance
I20260925 04:44:17.294556 393 external_mini_cluster.cc:1428] Setting key 5519ef57829c9db6c0c1e4d118dbce50
I20260925 04:44:17.296862 393 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 04:44:17.298707 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:17.291067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:44733 (local address 127.0.98.126:36341)
0925 04:44:17.291228 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:17.291232 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:17.291470 (+ 238us) server_negotiation.cc:407] Connection header received
0925 04:44:17.291669 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:17.291672 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:17.291718 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:17.291811 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:17.292413 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:17.292897 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:17.293748 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:17.293979 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:17.294091 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:17.295261 (+ 1170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:17.295264 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:17.295266 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:17.295287 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:17.296946 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:17.297438 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:17.297441 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:17.297442 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:17.297517 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:17.297811 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:17.297813 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:17.297814 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:17.297977 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:17.298088 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:17.298416 (+ 328us) server_negotiation.cc:299] Negotiation successful
0925 04:44:17.298544 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 04:44:17.298848 1997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:17.290980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:36341 (local address 127.0.98.67:44733)
0925 04:44:17.291379 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:17.291394 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:17.291527 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:17.291834 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:17.291836 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:17.292079 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:17.292280 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:17.292289 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:17.293034 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:17.293038 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:17.293625 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:17.293633 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:17.294087 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:17.294091 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:17.294208 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:17.294802 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:17.294823 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:17.295127 (+ 304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:44:17.297081 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:17.297084 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:17.297087 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:17.297338 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:17.297611 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:17.297614 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:17.297616 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:17.297718 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:17.298078 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:44:17.298084 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:44:17.298207 (+ 123us) client_negotiation.cc:769] Sending connection context
0925 04:44:17.298353 (+ 146us) client_negotiation.cc:240] Negotiation successful
0925 04:44:17.298535 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"mutex_wait_us":22,"thread_start_us":85,"threads_started":1}
Sep 25 04:44:17 dist-test-slave-kghz krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311457, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 04:44:17 dist-test-slave-kghz krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311457, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:44:17.323513 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:17.315344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48960 (local address 127.0.98.126:36341)
0925 04:44:17.315483 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:17.315487 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:17.315500 (+ 13us) server_negotiation.cc:407] Connection header received
0925 04:44:17.315543 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:17.315545 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:17.315585 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:17.315684 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:17.316387 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:17.316908 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:17.317766 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:17.317981 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:17.318075 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:17.320511 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:17.320514 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:17.320516 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:17.320545 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:17.321901 (+ 1356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:17.322393 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:17.322396 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:17.322397 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:17.322478 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:17.322742 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:17.322745 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:17.322746 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:17.322923 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:17.323025 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:17.323287 (+ 262us) server_negotiation.cc:299] Negotiation successful
0925 04:44:17.323390 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 04:44:17.326906 1507 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48960:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:44:17.329861 1507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:44:17.344295 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:17.338709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59336 (local address 127.0.98.65:43321)
0925 04:44:17.338971 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:17.338978 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:17.339442 (+ 464us) server_negotiation.cc:407] Connection header received
0925 04:44:17.339655 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:17.339661 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:17.339865 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:17.340126 (+ 261us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:44:17.340498 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:17.341376 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:17.342791 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:17.343605 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:17.343737 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:17.343897 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:17.344044 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 04:44:17.344137 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":74,"threads_started":1}
I20260925 04:44:17.344303 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:17.338566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:43321 (local address 127.0.0.1:59336)
0925 04:44:17.339287 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:17.339338 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:17.339481 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:17.340182 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:17.340189 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:17.340218 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:44:17.340377 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:17.340384 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:17.341485 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:17.341492 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:17.342658 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:17.342669 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:17.343737 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:17.343741 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:17.343855 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:17.343894 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 04:44:17.344014 (+ 120us) client_negotiation.cc:240] Negotiation successful
0925 04:44:17.344138 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":96,"threads_started":1}
I20260925 04:44:17.344909 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:17.339295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:37773 (local address 127.0.0.1:43042)
0925 04:44:17.340182 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:17.340194 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:17.340281 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:17.340711 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:17.340714 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:17.340732 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:44:17.340814 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:17.340819 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:17.342181 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:17.342184 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:17.343309 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:17.343321 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:17.344494 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:17.344498 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:17.344605 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:17.344630 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 04:44:17.344720 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 04:44:17.344786 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":835,"mutex_wait_us":32,"thread_start_us":90,"threads_started":1}
I20260925 04:44:17.344985 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:17.339683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43042 (local address 127.0.98.66:37773)
0925 04:44:17.340034 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:17.340067 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:17.340299 (+ 232us) server_negotiation.cc:407] Connection header received
0925 04:44:17.340383 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:17.340388 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:17.340577 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:17.340728 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:44:17.340919 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:17.342056 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:17.343444 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:17.344194 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:17.344286 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:17.344435 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:17.344760 (+ 325us) server_negotiation.cc:299] Negotiation successful
0925 04:44:17.344847 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":105,"threads_started":1}
I20260925 04:44:17.345137 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:17.338610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:46817 (local address 127.0.0.1:35794)
0925 04:44:17.339716 (+ 1106us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:17.339730 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:17.339810 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:17.340801 (+ 991us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:17.340804 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:17.340822 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:44:17.340935 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:17.340942 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:17.342439 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:17.342443 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:17.343669 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:17.343681 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:17.344771 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:17.344776 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:17.344873 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:17.344895 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 04:44:17.344944 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 04:44:17.345006 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1050,"mutex_wait_us":127,"thread_start_us":109,"threads_started":1}
I20260925 04:44:17.345904 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:17.339374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35794 (local address 127.0.98.67:46817)
0925 04:44:17.340406 (+ 1032us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:17.340417 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:17.340434 (+ 17us) server_negotiation.cc:407] Connection header received
0925 04:44:17.340494 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:17.340498 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:17.340655 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:17.341081 (+ 426us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:44:17.341218 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:17.342322 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:17.343801 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:17.344631 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:17.345458 (+ 827us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:17.345622 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:17.345670 (+ 48us) server_negotiation.cc:299] Negotiation successful
0925 04:44:17.345766 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":931,"thread_start_us":151,"threads_started":1}
I20260925 04:44:17.347641 1645 tablet_service.cc:1467] Processing CreateTablet for tablet b8de67e9dd1747ae8d07b831a3b51618 (DEFAULT_TABLE table=test-table [id=41e941123a804a34beeaeef7ee7eb9c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:44:17.347649 1785 tablet_service.cc:1467] Processing CreateTablet for tablet b8de67e9dd1747ae8d07b831a3b51618 (DEFAULT_TABLE table=test-table [id=41e941123a804a34beeaeef7ee7eb9c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:44:17.348435 1925 tablet_service.cc:1467] Processing CreateTablet for tablet b8de67e9dd1747ae8d07b831a3b51618 (DEFAULT_TABLE table=test-table [id=41e941123a804a34beeaeef7ee7eb9c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:44:17.348765 1645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8de67e9dd1747ae8d07b831a3b51618. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:44:17.348764 1785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8de67e9dd1747ae8d07b831a3b51618. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:44:17.349246 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8de67e9dd1747ae8d07b831a3b51618. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:44:17.356202 2014 tablet_bootstrap.cc:492] T b8de67e9dd1747ae8d07b831a3b51618 P 2f218b5c8db54a2ba8ecddc69387a36a: Bootstrap starting.
I20260925 04:44:17.357036 2015 tablet_bootstrap.cc:492] T b8de67e9dd1747ae8d07b831a3b51618 P 2a5a1853f59b4b63a2f6cd0c618f97f1: Bootstrap starting.
I20260925 04:44:17.357470 2016 tablet_bootstrap.cc:492] T b8de67e9dd1747ae8d07b831a3b51618 P 75fd7b5a529546e4b2428c240711a58b: Bootstrap starting.
I20260925 04:44:17.358520 2014 tablet_bootstrap.cc:654] T b8de67e9dd1747ae8d07b831a3b51618 P 2f218b5c8db54a2ba8ecddc69387a36a: Neither blocks nor log segments found. Creating new log.
I20260925 04:44:17.359239 2014 log.cc:824] T b8de67e9dd1747ae8d07b831a3b51618 P 2f218b5c8db54a2ba8ecddc69387a36a: Log is configured to *not* fsync() on all Append() calls
I20260925 04:44:17.359426 2016 tablet_bootstrap.cc:654] T b8de67e9dd1747ae8d07b831a3b51618 P 75fd7b5a529546e4b2428c240711a58b: Neither blocks nor log segments found. Creating new log.
I20260925 04:44:17.360183 2015 tablet_bootstrap.cc:654] T b8de67e9dd1747ae8d07b831a3b51618 P 2a5a1853f59b4b63a2f6cd0c618f97f1: Neither blocks nor log segments found. Creating new log.
I20260925 04:44:17.360244 2016 log.cc:824] T b8de67e9dd1747ae8d07b831a3b51618 P 75fd7b5a529546e4b2428c240711a58b: Log is configured to *not* fsync() on all Append() calls
I20260925 04:44:17.360971 2015 log.cc:824] T b8de67e9dd1747ae8d07b831a3b51618 P 2a5a1853f59b4b63a2f6cd0c618f97f1: Log is configured to *not* fsync() on all Append() calls
I20260925 04:44:17.361307 2014 tablet_bootstrap.cc:492] T b8de67e9dd1747ae8d07b831a3b51618 P 2f218b5c8db54a2ba8ecddc69387a36a: No bootstrap required, opened a new log
I20260925 04:44:17.361526 2014 ts_tablet_manager.cc:1397] T b8de67e9dd1747ae8d07b831a3b51618 P 2f218b5c8db54a2ba8ecddc69387a36a: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 04:44:17.362272 2016 tablet_bootstrap.cc:492] T b8de67e9dd1747ae8d07b831a3b51618 P 75fd7b5a529546e4b2428c240711a58b: No bootstrap required, opened a new log
I20260925 04:44:17.362438 2016 ts_tablet_manager.cc:1397] T b8de67e9dd1747ae8d07b831a3b51618 P 75fd7b5a529546e4b2428c240711a58b: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260925 04:44:17.363183 2015 tablet_bootstrap.cc:492] T b8de67e9dd1747ae8d07b831a3b51618 P 2a5a1853f59b4b63a2f6cd0c618f97f1: No bootstrap required, opened a new log
I20260925 04:44:17.363363 2015 ts_tablet_manager.cc:1397] T b8de67e9dd1747ae8d07b831a3b51618 P 2a5a1853f59b4b63a2f6cd0c618f97f1: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 04:44:17.365621 2014 raft_consensus.cc:348] T b8de67e9dd1747ae8d07b831a3b51618 P 2f218b5c8db54a2ba8ecddc69387a36a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a5a1853f59b4b63a2f6cd0c618f97f1" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43321 } } peers { permanent_uuid: "2f218b5c8db54a2ba8ecddc69387a36a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 46817 } } peers { permanent_uuid: "75fd7b5a529546e4b2428c240711a58b" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37773 } }
I20260925 04:44:17.365900 2014 raft_consensus.cc:374] T b8de67e9dd1747ae8d07b831a3b51618 P 2f218b5c8db54a2ba8ecddc69387a36a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:44:17.365986 2014 raft_consensus.cc:729] T b8de67e9dd1747ae8d07b831a3b51618 P 2f218b5c8db54a2ba8ecddc69387a36a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f218b5c8db54a2ba8ecddc69387a36a, State: Initialized, Role: FOLLOWER
I20260925 04:44:17.366284 2016 raft_consensus.cc:348] T b8de67e9dd1747ae8d07b831a3b51618 P 75fd7b5a529546e4b2428c240711a58b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a5a1853f59b4b63a2f6cd0c618f97f1" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43321 } } peers { permanent_uuid: "2f218b5c8db54a2ba8ecddc69387a36a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 46817 } } peers { permanent_uuid: "75fd7b5a529546e4b2428c240711a58b" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37773 } }
I20260925 04:44:17.366544 2016 raft_consensus.cc:374] T b8de67e9dd1747ae8d07b831a3b51618 P 75fd7b5a529546e4b2428c240711a58b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:44:17.366465 2014 consensus_queue.cc:260] T b8de67e9dd1747ae8d07b831a3b51618 P 2f218b5c8db54a2ba8ecddc69387a36a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a5a1853f59b4b63a2f6cd0c618f97f1" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43321 } } peers { permanent_uuid: "2f218b5c8db54a2ba8ecddc69387a36a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 46817 } } peers { permanent_uuid: "75fd7b5a529546e4b2428c240711a58b" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37773 } }
I20260925 04:44:17.366627 2016 raft_consensus.cc:729] T b8de67e9dd1747ae8d07b831a3b51618 P 75fd7b5a529546e4b2428c240711a58b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75fd7b5a529546e4b2428c240711a58b, State: Initialized, Role: FOLLOWER
I20260925 04:44:17.366474 2015 raft_consensus.cc:348] T b8de67e9dd1747ae8d07b831a3b51618 P 2a5a1853f59b4b63a2f6cd0c618f97f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a5a1853f59b4b63a2f6cd0c618f97f1" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43321 } } peers { permanent_uuid: "2f218b5c8db54a2ba8ecddc69387a36a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 46817 } } peers { permanent_uuid: "75fd7b5a529546e4b2428c240711a58b" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37773 } }
I20260925 04:44:17.366755 2015 raft_consensus.cc:374] T b8de67e9dd1747ae8d07b831a3b51618 P 2a5a1853f59b4b63a2f6cd0c618f97f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:44:17.366815 2015 raft_consensus.cc:729] T b8de67e9dd1747ae8d07b831a3b51618 P 2a5a1853f59b4b63a2f6cd0c618f97f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a5a1853f59b4b63a2f6cd0c618f97f1, State: Initialized, Role: FOLLOWER
I20260925 04:44:17.367096 2016 consensus_queue.cc:260] T b8de67e9dd1747ae8d07b831a3b51618 P 75fd7b5a529546e4b2428c240711a58b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a5a1853f59b4b63a2f6cd0c618f97f1" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43321 } } peers { permanent_uuid: "2f218b5c8db54a2ba8ecddc69387a36a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 46817 } } peers { permanent_uuid: "75fd7b5a529546e4b2428c240711a58b" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37773 } }
I20260925 04:44:17.367285 2015 consensus_queue.cc:260] T b8de67e9dd1747ae8d07b831a3b51618 P 2a5a1853f59b4b63a2f6cd0c618f97f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a5a1853f59b4b63a2f6cd0c618f97f1" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43321 } } peers { permanent_uuid: "2f218b5c8db54a2ba8ecddc69387a36a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 46817 } } peers { permanent_uuid: "75fd7b5a529546e4b2428c240711a58b" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37773 } }
I20260925 04:44:17.367501 2014 ts_tablet_manager.cc:1428] T b8de67e9dd1747ae8d07b831a3b51618 P 2f218b5c8db54a2ba8ecddc69387a36a: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260925 04:44:17.367810 1851 heartbeater.cc:503] Master 127.0.98.126:36341 was elected leader, sending a full tablet report...
I20260925 04:44:17.368036 2016 ts_tablet_manager.cc:1428] T b8de67e9dd1747ae8d07b831a3b51618 P 75fd7b5a529546e4b2428c240711a58b: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260925 04:44:17.368413 2015 ts_tablet_manager.cc:1428] T b8de67e9dd1747ae8d07b831a3b51618 P 2a5a1853f59b4b63a2f6cd0c618f97f1: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260925 04:44:17.368597 1711 heartbeater.cc:503] Master 127.0.98.126:36341 was elected leader, sending a full tablet report...
I20260925 04:44:17.369028 1991 heartbeater.cc:503] Master 127.0.98.126:36341 was elected leader, sending a full tablet report...
I20260925 04:44:17.442849 2020 raft_consensus.cc:482] T b8de67e9dd1747ae8d07b831a3b51618 P 2f218b5c8db54a2ba8ecddc69387a36a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:44:17.443099 2020 raft_consensus.cc:504] T b8de67e9dd1747ae8d07b831a3b51618 P 2f218b5c8db54a2ba8ecddc69387a36a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a5a1853f59b4b63a2f6cd0c618f97f1" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43321 } } peers { permanent_uuid: "2f218b5c8db54a2ba8ecddc69387a36a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 46817 } } peers { permanent_uuid: "75fd7b5a529546e4b2428c240711a58b" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37773 } }
I20260925 04:44:17.444316 2020 leader_election.cc:288] T b8de67e9dd1747ae8d07b831a3b51618 P 2f218b5c8db54a2ba8ecddc69387a36a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a5a1853f59b4b63a2f6cd0c618f97f1 (127.0.98.65:43321), 75fd7b5a529546e4b2428c240711a58b (127.0.98.66:37773)
I20260925 04:44:17.448195 1884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:17.444710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:43321 (local address 127.0.98.67:57837)
0925 04:44:17.444905 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:17.444924 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:17.445042 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:17.445386 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:17.445389 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:17.445408 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:44:17.445535 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:17.445540 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:17.446293 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:17.446298 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:17.447090 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:17.447098 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:17.447752 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:17.447755 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:17.447856 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:17.447877 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 04:44:17.447980 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 04:44:17.448064 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260925 04:44:17.448195 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:17.444808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:57837 (local address 127.0.98.65:43321)
0925 04:44:17.444977 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:17.444982 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:17.445052 (+ 70us) server_negotiation.cc:407] Connection header received
0925 04:44:17.445176 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:17.445182 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:17.445247 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:17.445334 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:44:17.445654 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:17.446158 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:17.447218 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:17.447631 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:17.447747 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:17.447810 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:17.447995 (+ 185us) server_negotiation.cc:299] Negotiation successful
0925 04:44:17.448064 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 04:44:17.448921 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8de67e9dd1747ae8d07b831a3b51618" candidate_uuid: "2f218b5c8db54a2ba8ecddc69387a36a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a5a1853f59b4b63a2f6cd0c618f97f1" is_pre_election: true
I20260925 04:44:17.449280 1665 raft_consensus.cc:2454] T b8de67e9dd1747ae8d07b831a3b51618 P 2a5a1853f59b4b63a2f6cd0c618f97f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f218b5c8db54a2ba8ecddc69387a36a in term 0.
I20260925 04:44:17.449831 1879 leader_election.cc:302] T b8de67e9dd1747ae8d07b831a3b51618 P 2f218b5c8db54a2ba8ecddc69387a36a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a5a1853f59b4b63a2f6cd0c618f97f1, 2f218b5c8db54a2ba8ecddc69387a36a; no voters:
I20260925 04:44:17.449877 2023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:17.445078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:37773 (local address 127.0.98.67:39627)
0925 04:44:17.445434 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:17.445445 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:17.445575 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:17.445841 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:17.445844 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:17.445859 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:44:17.445999 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:17.446004 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:17.446830 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:17.446837 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:17.448516 (+ 1679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:17.448525 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:17.449426 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:17.449431 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:17.449536 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:17.449591 (+ 55us) client_negotiation.cc:769] Sending connection context
0925 04:44:17.449675 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 04:44:17.449752 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":132,"threads_started":1}
I20260925 04:44:17.450111 2020 raft_consensus.cc:2781] T b8de67e9dd1747ae8d07b831a3b51618 P 2f218b5c8db54a2ba8ecddc69387a36a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:44:17.450209 2020 raft_consensus.cc:482] T b8de67e9dd1747ae8d07b831a3b51618 P 2f218b5c8db54a2ba8ecddc69387a36a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:44:17.450277 2020 raft_consensus.cc:3037] T b8de67e9dd1747ae8d07b831a3b51618 P 2f218b5c8db54a2ba8ecddc69387a36a [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:44:17.450309 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:17.445203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:39627 (local address 127.0.98.66:37773)
0925 04:44:17.445341 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:17.445345 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:17.445537 (+ 192us) server_negotiation.cc:407] Connection header received
0925 04:44:17.445688 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:17.445691 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:17.445744 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:17.445829 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:44:17.446106 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:17.446717 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:17.448695 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:17.449245 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:17.449360 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:17.449421 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:17.450075 (+ 654us) server_negotiation.cc:299] Negotiation successful
0925 04:44:17.450162 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 04:44:17.450881 1805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8de67e9dd1747ae8d07b831a3b51618" candidate_uuid: "2f218b5c8db54a2ba8ecddc69387a36a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75fd7b5a529546e4b2428c240711a58b" is_pre_election: true
I20260925 04:44:17.451203 1805 raft_consensus.cc:2454] T b8de67e9dd1747ae8d07b831a3b51618 P 75fd7b5a529546e4b2428c240711a58b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f218b5c8db54a2ba8ecddc69387a36a in term 0.
I20260925 04:44:17.451659 2020 raft_consensus.cc:504] T b8de67e9dd1747ae8d07b831a3b51618 P 2f218b5c8db54a2ba8ecddc69387a36a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a5a1853f59b4b63a2f6cd0c618f97f1" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43321 } } peers { permanent_uuid: "2f218b5c8db54a2ba8ecddc69387a36a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 46817 } } peers { permanent_uuid: "75fd7b5a529546e4b2428c240711a58b" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37773 } }
I20260925 04:44:17.452003 2020 leader_election.cc:288] T b8de67e9dd1747ae8d07b831a3b51618 P 2f218b5c8db54a2ba8ecddc69387a36a [CANDIDATE]: Term 1 election: Requested vote from peers 2a5a1853f59b4b63a2f6cd0c618f97f1 (127.0.98.65:43321), 75fd7b5a529546e4b2428c240711a58b (127.0.98.66:37773)
I20260925 04:44:17.452338 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8de67e9dd1747ae8d07b831a3b51618" candidate_uuid: "2f218b5c8db54a2ba8ecddc69387a36a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a5a1853f59b4b63a2f6cd0c618f97f1"
I20260925 04:44:17.452473 1805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8de67e9dd1747ae8d07b831a3b51618" candidate_uuid: "2f218b5c8db54a2ba8ecddc69387a36a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75fd7b5a529546e4b2428c240711a58b"
I20260925 04:44:17.452479 1665 raft_consensus.cc:3037] T b8de67e9dd1747ae8d07b831a3b51618 P 2a5a1853f59b4b63a2f6cd0c618f97f1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:44:17.452593 1805 raft_consensus.cc:3037] T b8de67e9dd1747ae8d07b831a3b51618 P 75fd7b5a529546e4b2428c240711a58b [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:44:17.453888 1665 raft_consensus.cc:2454] T b8de67e9dd1747ae8d07b831a3b51618 P 2a5a1853f59b4b63a2f6cd0c618f97f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f218b5c8db54a2ba8ecddc69387a36a in term 1.
I20260925 04:44:17.454339 1805 raft_consensus.cc:2454] T b8de67e9dd1747ae8d07b831a3b51618 P 75fd7b5a529546e4b2428c240711a58b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f218b5c8db54a2ba8ecddc69387a36a in term 1.
I20260925 04:44:17.454409 1879 leader_election.cc:302] T b8de67e9dd1747ae8d07b831a3b51618 P 2f218b5c8db54a2ba8ecddc69387a36a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a5a1853f59b4b63a2f6cd0c618f97f1, 2f218b5c8db54a2ba8ecddc69387a36a; no voters:
I20260925 04:44:17.454619 2020 raft_consensus.cc:2781] T b8de67e9dd1747ae8d07b831a3b51618 P 2f218b5c8db54a2ba8ecddc69387a36a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:44:17.454842 2020 raft_consensus.cc:686] T b8de67e9dd1747ae8d07b831a3b51618 P 2f218b5c8db54a2ba8ecddc69387a36a [term 1 LEADER]: Becoming Leader. State: Replica: 2f218b5c8db54a2ba8ecddc69387a36a, State: Running, Role: LEADER
I20260925 04:44:17.455209 2020 consensus_queue.cc:237] T b8de67e9dd1747ae8d07b831a3b51618 P 2f218b5c8db54a2ba8ecddc69387a36a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a5a1853f59b4b63a2f6cd0c618f97f1" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43321 } } peers { permanent_uuid: "2f218b5c8db54a2ba8ecddc69387a36a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 46817 } } peers { permanent_uuid: "75fd7b5a529546e4b2428c240711a58b" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37773 } }
I20260925 04:44:17.458654 1506 catalog_manager.cc:5495] T b8de67e9dd1747ae8d07b831a3b51618 P 2f218b5c8db54a2ba8ecddc69387a36a reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f218b5c8db54a2ba8ecddc69387a36a (127.0.98.67). New cstate: current_term: 1 leader_uuid: "2f218b5c8db54a2ba8ecddc69387a36a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a5a1853f59b4b63a2f6cd0c618f97f1" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f218b5c8db54a2ba8ecddc69387a36a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 46817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "75fd7b5a529546e4b2428c240711a58b" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37773 } health_report { overall_health: UNKNOWN } } }
I20260925 04:44:17.502892 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:17.499014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35810 (local address 127.0.98.67:46817)
0925 04:44:17.499363 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:17.499367 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:17.499381 (+ 14us) server_negotiation.cc:407] Connection header received
0925 04:44:17.499447 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:17.499449 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:17.499504 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:17.499657 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:44:17.499962 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:17.500474 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:17.501359 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:17.501586 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:17.501698 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:17.502031 (+ 333us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:44:17.502490 (+ 459us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:44:17.502614 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:17.502689 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 04:44:17.502744 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213}
I20260925 04:44:17.508615 1665 raft_consensus.cc:1261] T b8de67e9dd1747ae8d07b831a3b51618 P 2a5a1853f59b4b63a2f6cd0c618f97f1 [term 1 FOLLOWER]: Refusing update from remote peer 2f218b5c8db54a2ba8ecddc69387a36a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:44:17.508615 1805 raft_consensus.cc:1261] T b8de67e9dd1747ae8d07b831a3b51618 P 75fd7b5a529546e4b2428c240711a58b [term 1 FOLLOWER]: Refusing update from remote peer 2f218b5c8db54a2ba8ecddc69387a36a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:44:17.509358 2020 consensus_queue.cc:1035] T b8de67e9dd1747ae8d07b831a3b51618 P 2f218b5c8db54a2ba8ecddc69387a36a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a5a1853f59b4b63a2f6cd0c618f97f1" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:44:17.509615 2024 consensus_queue.cc:1035] T b8de67e9dd1747ae8d07b831a3b51618 P 2f218b5c8db54a2ba8ecddc69387a36a [LEADER]: Connected to new peer: Peer: permanent_uuid: "75fd7b5a529546e4b2428c240711a58b" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:44:17.520663 2028 mvcc.cc:205] Tried to move back new op lower bound from 7333115729947992064 to 7333115729739669504. Current Snapshot: MvccSnapshot[applied={T|T < 7333115729947992064 or (T in {7333115729947992064})}]
I20260925 04:44:17.520690 2029 mvcc.cc:205] Tried to move back new op lower bound from 7333115729947992064 to 7333115729739669504. Current Snapshot: MvccSnapshot[applied={T|T < 7333115729947992064 or (T in {7333115729947992064})}]
I20260925 04:44:17.531647 1507 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48960:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:44:17.531836 1507 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48960:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:44:17.534592 1507 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6f7507ec830a4633868cda169231b809: Sending DeleteTablet for 3 replicas of tablet b8de67e9dd1747ae8d07b831a3b51618
I20260925 04:44:17.535440 1785 tablet_service.cc:1514] Processing DeleteTablet for tablet b8de67e9dd1747ae8d07b831a3b51618 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:44:17 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:43042
I20260925 04:44:17.535506 1925 tablet_service.cc:1514] Processing DeleteTablet for tablet b8de67e9dd1747ae8d07b831a3b51618 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:44:17 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:35794
I20260925 04:44:17.535477 1645 tablet_service.cc:1514] Processing DeleteTablet for tablet b8de67e9dd1747ae8d07b831a3b51618 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:44:17 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:59336
I20260925 04:44:17.536103 2037 tablet_replica.cc:330] T b8de67e9dd1747ae8d07b831a3b51618 P 75fd7b5a529546e4b2428c240711a58b: stopping tablet replica
I20260925 04:44:17.536464 2037 raft_consensus.cc:2229] T b8de67e9dd1747ae8d07b831a3b51618 P 75fd7b5a529546e4b2428c240711a58b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:44:17.536870 1507 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48960:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:44:17.537313 1507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:44:17.538174 2037 raft_consensus.cc:2258] T b8de67e9dd1747ae8d07b831a3b51618 P 75fd7b5a529546e4b2428c240711a58b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:44:17.538688 2038 tablet_replica.cc:330] T b8de67e9dd1747ae8d07b831a3b51618 P 2f218b5c8db54a2ba8ecddc69387a36a: stopping tablet replica
I20260925 04:44:17.540643 2038 raft_consensus.cc:2229] T b8de67e9dd1747ae8d07b831a3b51618 P 2f218b5c8db54a2ba8ecddc69387a36a [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:44:17.541877 1785 tablet_service.cc:1467] Processing CreateTablet for tablet e3cf9e5729dd4ee0b53ab7218f152449 (DEFAULT_TABLE table=test-table [id=df7775d6e90745c0b794d5599ab149da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:44:17.542162 1785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3cf9e5729dd4ee0b53ab7218f152449. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:44:17.541991 1925 tablet_service.cc:1467] Processing CreateTablet for tablet e3cf9e5729dd4ee0b53ab7218f152449 (DEFAULT_TABLE table=test-table [id=df7775d6e90745c0b794d5599ab149da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:44:17.542966 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3cf9e5729dd4ee0b53ab7218f152449. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:44:17.544883 2016 tablet_bootstrap.cc:492] T e3cf9e5729dd4ee0b53ab7218f152449 P 75fd7b5a529546e4b2428c240711a58b: Bootstrap starting.
I20260925 04:44:17.545588 1645 tablet_service.cc:1467] Processing CreateTablet for tablet e3cf9e5729dd4ee0b53ab7218f152449 (DEFAULT_TABLE table=test-table [id=df7775d6e90745c0b794d5599ab149da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:44:17.545846 1645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3cf9e5729dd4ee0b53ab7218f152449. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:44:17.545933 2016 tablet_bootstrap.cc:654] T e3cf9e5729dd4ee0b53ab7218f152449 P 75fd7b5a529546e4b2428c240711a58b: Neither blocks nor log segments found. Creating new log.
I20260925 04:44:17.546680 2014 tablet_bootstrap.cc:492] T e3cf9e5729dd4ee0b53ab7218f152449 P 2f218b5c8db54a2ba8ecddc69387a36a: Bootstrap starting.
I20260925 04:44:17.547505 2038 raft_consensus.cc:2258] T b8de67e9dd1747ae8d07b831a3b51618 P 2f218b5c8db54a2ba8ecddc69387a36a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:44:17.547746 2037 ts_tablet_manager.cc:1905] T b8de67e9dd1747ae8d07b831a3b51618 P 75fd7b5a529546e4b2428c240711a58b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:44:17.547590 2014 tablet_bootstrap.cc:654] T e3cf9e5729dd4ee0b53ab7218f152449 P 2f218b5c8db54a2ba8ecddc69387a36a: Neither blocks nor log segments found. Creating new log.
I20260925 04:44:17.547917 2039 tablet_replica.cc:330] T b8de67e9dd1747ae8d07b831a3b51618 P 2a5a1853f59b4b63a2f6cd0c618f97f1: stopping tablet replica
I20260925 04:44:17.548250 2015 tablet_bootstrap.cc:492] T e3cf9e5729dd4ee0b53ab7218f152449 P 2a5a1853f59b4b63a2f6cd0c618f97f1: Bootstrap starting.
I20260925 04:44:17.548278 2039 raft_consensus.cc:2229] T b8de67e9dd1747ae8d07b831a3b51618 P 2a5a1853f59b4b63a2f6cd0c618f97f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:44:17.548549 2039 raft_consensus.cc:2258] T b8de67e9dd1747ae8d07b831a3b51618 P 2a5a1853f59b4b63a2f6cd0c618f97f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:44:17.549216 2015 tablet_bootstrap.cc:654] T e3cf9e5729dd4ee0b53ab7218f152449 P 2a5a1853f59b4b63a2f6cd0c618f97f1: Neither blocks nor log segments found. Creating new log.
I20260925 04:44:17.549862 2014 tablet_bootstrap.cc:492] T e3cf9e5729dd4ee0b53ab7218f152449 P 2f218b5c8db54a2ba8ecddc69387a36a: No bootstrap required, opened a new log
I20260925 04:44:17.549942 2014 ts_tablet_manager.cc:1397] T e3cf9e5729dd4ee0b53ab7218f152449 P 2f218b5c8db54a2ba8ecddc69387a36a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 04:44:17.550148 2039 ts_tablet_manager.cc:1905] T b8de67e9dd1747ae8d07b831a3b51618 P 2a5a1853f59b4b63a2f6cd0c618f97f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:44:17.550426 2014 raft_consensus.cc:348] T e3cf9e5729dd4ee0b53ab7218f152449 P 2f218b5c8db54a2ba8ecddc69387a36a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a5a1853f59b4b63a2f6cd0c618f97f1" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43321 } } peers { permanent_uuid: "75fd7b5a529546e4b2428c240711a58b" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37773 } } peers { permanent_uuid: "2f218b5c8db54a2ba8ecddc69387a36a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 46817 } }
I20260925 04:44:17.550570 2014 raft_consensus.cc:374] T e3cf9e5729dd4ee0b53ab7218f152449 P 2f218b5c8db54a2ba8ecddc69387a36a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:44:17.550606 2014 raft_consensus.cc:729] T e3cf9e5729dd4ee0b53ab7218f152449 P 2f218b5c8db54a2ba8ecddc69387a36a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f218b5c8db54a2ba8ecddc69387a36a, State: Initialized, Role: FOLLOWER
I20260925 04:44:17.550643 2037 ts_tablet_manager.cc:1918] T b8de67e9dd1747ae8d07b831a3b51618 P 75fd7b5a529546e4b2428c240711a58b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:44:17.550732 2037 log.cc:1196] T b8de67e9dd1747ae8d07b831a3b51618 P 75fd7b5a529546e4b2428c240711a58b: Deleting WAL directory at /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/minicluster-data/ts-1/wal/wals/b8de67e9dd1747ae8d07b831a3b51618
I20260925 04:44:17.550740 2014 consensus_queue.cc:260] T e3cf9e5729dd4ee0b53ab7218f152449 P 2f218b5c8db54a2ba8ecddc69387a36a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a5a1853f59b4b63a2f6cd0c618f97f1" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43321 } } peers { permanent_uuid: "75fd7b5a529546e4b2428c240711a58b" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37773 } } peers { permanent_uuid: "2f218b5c8db54a2ba8ecddc69387a36a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 46817 } }
I20260925 04:44:17.551065 2037 ts_tablet_manager.cc:1939] T b8de67e9dd1747ae8d07b831a3b51618 P 75fd7b5a529546e4b2428c240711a58b: Deleting consensus metadata
I20260925 04:44:17.551061 2014 ts_tablet_manager.cc:1428] T e3cf9e5729dd4ee0b53ab7218f152449 P 2f218b5c8db54a2ba8ecddc69387a36a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 04:44:17.552611 2039 ts_tablet_manager.cc:1918] T b8de67e9dd1747ae8d07b831a3b51618 P 2a5a1853f59b4b63a2f6cd0c618f97f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:44:17.552738 2039 log.cc:1196] T b8de67e9dd1747ae8d07b831a3b51618 P 2a5a1853f59b4b63a2f6cd0c618f97f1: Deleting WAL directory at /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/minicluster-data/ts-0/wal/wals/b8de67e9dd1747ae8d07b831a3b51618
I20260925 04:44:17.552872 2016 tablet_bootstrap.cc:492] T e3cf9e5729dd4ee0b53ab7218f152449 P 75fd7b5a529546e4b2428c240711a58b: No bootstrap required, opened a new log
I20260925 04:44:17.552959 2016 ts_tablet_manager.cc:1397] T e3cf9e5729dd4ee0b53ab7218f152449 P 75fd7b5a529546e4b2428c240711a58b: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.002s
I20260925 04:44:17.553083 2039 ts_tablet_manager.cc:1939] T b8de67e9dd1747ae8d07b831a3b51618 P 2a5a1853f59b4b63a2f6cd0c618f97f1: Deleting consensus metadata
I20260925 04:44:17.553092 2015 tablet_bootstrap.cc:492] T e3cf9e5729dd4ee0b53ab7218f152449 P 2a5a1853f59b4b63a2f6cd0c618f97f1: No bootstrap required, opened a new log
I20260925 04:44:17.553203 2015 ts_tablet_manager.cc:1397] T e3cf9e5729dd4ee0b53ab7218f152449 P 2a5a1853f59b4b63a2f6cd0c618f97f1: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260925 04:44:17.553519 2016 raft_consensus.cc:348] T e3cf9e5729dd4ee0b53ab7218f152449 P 75fd7b5a529546e4b2428c240711a58b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a5a1853f59b4b63a2f6cd0c618f97f1" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43321 } } peers { permanent_uuid: "75fd7b5a529546e4b2428c240711a58b" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37773 } } peers { permanent_uuid: "2f218b5c8db54a2ba8ecddc69387a36a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 46817 } }
I20260925 04:44:17.553649 2016 raft_consensus.cc:374] T e3cf9e5729dd4ee0b53ab7218f152449 P 75fd7b5a529546e4b2428c240711a58b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:44:17.553630 1495 catalog_manager.cc:4841] TS 75fd7b5a529546e4b2428c240711a58b (127.0.98.66:37773): tablet b8de67e9dd1747ae8d07b831a3b51618 (table test-table [id=41e941123a804a34beeaeef7ee7eb9c8]) successfully deleted
I20260925 04:44:17.553684 2016 raft_consensus.cc:729] T e3cf9e5729dd4ee0b53ab7218f152449 P 75fd7b5a529546e4b2428c240711a58b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75fd7b5a529546e4b2428c240711a58b, State: Initialized, Role: FOLLOWER
I20260925 04:44:17.553800 2016 consensus_queue.cc:260] T e3cf9e5729dd4ee0b53ab7218f152449 P 75fd7b5a529546e4b2428c240711a58b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a5a1853f59b4b63a2f6cd0c618f97f1" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43321 } } peers { permanent_uuid: "75fd7b5a529546e4b2428c240711a58b" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37773 } } peers { permanent_uuid: "2f218b5c8db54a2ba8ecddc69387a36a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 46817 } }
I20260925 04:44:17.553855 2015 raft_consensus.cc:348] T e3cf9e5729dd4ee0b53ab7218f152449 P 2a5a1853f59b4b63a2f6cd0c618f97f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a5a1853f59b4b63a2f6cd0c618f97f1" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43321 } } peers { permanent_uuid: "75fd7b5a529546e4b2428c240711a58b" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37773 } } peers { permanent_uuid: "2f218b5c8db54a2ba8ecddc69387a36a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 46817 } }
I20260925 04:44:17.553946 2015 raft_consensus.cc:374] T e3cf9e5729dd4ee0b53ab7218f152449 P 2a5a1853f59b4b63a2f6cd0c618f97f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:44:17.553977 2015 raft_consensus.cc:729] T e3cf9e5729dd4ee0b53ab7218f152449 P 2a5a1853f59b4b63a2f6cd0c618f97f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a5a1853f59b4b63a2f6cd0c618f97f1, State: Initialized, Role: FOLLOWER
I20260925 04:44:17.554064 2016 ts_tablet_manager.cc:1428] T e3cf9e5729dd4ee0b53ab7218f152449 P 75fd7b5a529546e4b2428c240711a58b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260925 04:44:17.554105 2015 consensus_queue.cc:260] T e3cf9e5729dd4ee0b53ab7218f152449 P 2a5a1853f59b4b63a2f6cd0c618f97f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a5a1853f59b4b63a2f6cd0c618f97f1" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43321 } } peers { permanent_uuid: "75fd7b5a529546e4b2428c240711a58b" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37773 } } peers { permanent_uuid: "2f218b5c8db54a2ba8ecddc69387a36a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 46817 } }
I20260925 04:44:17.554355 1495 catalog_manager.cc:4841] TS 2a5a1853f59b4b63a2f6cd0c618f97f1 (127.0.98.65:43321): tablet b8de67e9dd1747ae8d07b831a3b51618 (table test-table [id=41e941123a804a34beeaeef7ee7eb9c8]) successfully deleted
I20260925 04:44:17.555266 2038 ts_tablet_manager.cc:1905] T b8de67e9dd1747ae8d07b831a3b51618 P 2f218b5c8db54a2ba8ecddc69387a36a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:44:17.555356 2015 ts_tablet_manager.cc:1428] T e3cf9e5729dd4ee0b53ab7218f152449 P 2a5a1853f59b4b63a2f6cd0c618f97f1: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260925 04:44:17.557996 2038 ts_tablet_manager.cc:1918] T b8de67e9dd1747ae8d07b831a3b51618 P 2f218b5c8db54a2ba8ecddc69387a36a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:44:17.558080 2038 log.cc:1196] T b8de67e9dd1747ae8d07b831a3b51618 P 2f218b5c8db54a2ba8ecddc69387a36a: Deleting WAL directory at /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/minicluster-data/ts-2/wal/wals/b8de67e9dd1747ae8d07b831a3b51618
I20260925 04:44:17.558436 2038 ts_tablet_manager.cc:1939] T b8de67e9dd1747ae8d07b831a3b51618 P 2f218b5c8db54a2ba8ecddc69387a36a: Deleting consensus metadata
W20260925 04:44:17.559157 1852 tablet.cc:2369] T e3cf9e5729dd4ee0b53ab7218f152449 P 75fd7b5a529546e4b2428c240711a58b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:44:17.559535 1492 catalog_manager.cc:4841] TS 2f218b5c8db54a2ba8ecddc69387a36a (127.0.98.67:46817): tablet b8de67e9dd1747ae8d07b831a3b51618 (table test-table [id=41e941123a804a34beeaeef7ee7eb9c8]) successfully deleted
W20260925 04:44:17.581059 1712 tablet.cc:2369] T e3cf9e5729dd4ee0b53ab7218f152449 P 2a5a1853f59b4b63a2f6cd0c618f97f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:44:17.703320 2022 raft_consensus.cc:482] T e3cf9e5729dd4ee0b53ab7218f152449 P 2a5a1853f59b4b63a2f6cd0c618f97f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:44:17.703467 2022 raft_consensus.cc:504] T e3cf9e5729dd4ee0b53ab7218f152449 P 2a5a1853f59b4b63a2f6cd0c618f97f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a5a1853f59b4b63a2f6cd0c618f97f1" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43321 } } peers { permanent_uuid: "75fd7b5a529546e4b2428c240711a58b" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37773 } } peers { permanent_uuid: "2f218b5c8db54a2ba8ecddc69387a36a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 46817 } }
I20260925 04:44:17.704557 2022 leader_election.cc:288] T e3cf9e5729dd4ee0b53ab7218f152449 P 2a5a1853f59b4b63a2f6cd0c618f97f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 75fd7b5a529546e4b2428c240711a58b (127.0.98.66:37773), 2f218b5c8db54a2ba8ecddc69387a36a (127.0.98.67:46817)
I20260925 04:44:17.708707 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:17.704966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:54911 (local address 127.0.98.67:46817)
0925 04:44:17.705104 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:17.705109 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:17.705420 (+ 311us) server_negotiation.cc:407] Connection header received
0925 04:44:17.705554 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:17.705558 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:17.705604 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:17.705683 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:44:17.706032 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:17.706575 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:17.707724 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:17.708111 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:17.708238 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:17.708308 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:17.708518 (+ 210us) server_negotiation.cc:299] Negotiation successful
0925 04:44:17.708588 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 04:44:17.708709 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:17.704956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:34713 (local address 127.0.98.66:37773)
0925 04:44:17.705094 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:17.705099 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:17.705257 (+ 158us) server_negotiation.cc:407] Connection header received
0925 04:44:17.705361 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:17.705367 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:17.705425 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:17.705501 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:44:17.705870 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:17.706533 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:17.707664 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:17.708094 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:17.708176 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:17.708277 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:17.708484 (+ 207us) server_negotiation.cc:299] Negotiation successful
0925 04:44:17.708554 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 04:44:17.708738 2043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:17.704809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:37773 (local address 127.0.98.65:34713)
0925 04:44:17.705149 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:17.705164 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:17.705269 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:17.705540 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:17.705543 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:17.705578 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:44:17.705723 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:17.705731 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:17.706657 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:17.706663 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:17.707511 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:17.707519 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:17.708229 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:17.708233 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:17.708348 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:17.708375 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 04:44:17.708437 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 04:44:17.708531 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":11,"thread_start_us":89,"threads_started":1}
I20260925 04:44:17.708742 2044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:17.704882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:46817 (local address 127.0.98.65:54911)
0925 04:44:17.705308 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:17.705323 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:17.705430 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:17.705728 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:17.705731 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:17.705747 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:44:17.705910 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:17.705916 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:17.706740 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:17.706743 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:17.707562 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:17.707571 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:17.708285 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:17.708291 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:17.708375 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:17.708392 (+ 17us) client_negotiation.cc:769] Sending connection context
0925 04:44:17.708486 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 04:44:17.708554 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"mutex_wait_us":21,"thread_start_us":142,"threads_started":1}
I20260925 04:44:17.709183 1805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3cf9e5729dd4ee0b53ab7218f152449" candidate_uuid: "2a5a1853f59b4b63a2f6cd0c618f97f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75fd7b5a529546e4b2428c240711a58b" is_pre_election: true
I20260925 04:44:17.709362 1805 raft_consensus.cc:2454] T e3cf9e5729dd4ee0b53ab7218f152449 P 75fd7b5a529546e4b2428c240711a58b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a5a1853f59b4b63a2f6cd0c618f97f1 in term 0.
I20260925 04:44:17.709512 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3cf9e5729dd4ee0b53ab7218f152449" candidate_uuid: "2a5a1853f59b4b63a2f6cd0c618f97f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f218b5c8db54a2ba8ecddc69387a36a" is_pre_election: true
I20260925 04:44:17.709756 1945 raft_consensus.cc:2454] T e3cf9e5729dd4ee0b53ab7218f152449 P 2f218b5c8db54a2ba8ecddc69387a36a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a5a1853f59b4b63a2f6cd0c618f97f1 in term 0.
I20260925 04:44:17.709766 1599 leader_election.cc:302] T e3cf9e5729dd4ee0b53ab7218f152449 P 2a5a1853f59b4b63a2f6cd0c618f97f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a5a1853f59b4b63a2f6cd0c618f97f1, 75fd7b5a529546e4b2428c240711a58b; no voters:
I20260925 04:44:17.710011 2022 raft_consensus.cc:2781] T e3cf9e5729dd4ee0b53ab7218f152449 P 2a5a1853f59b4b63a2f6cd0c618f97f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:44:17.710134 2022 raft_consensus.cc:482] T e3cf9e5729dd4ee0b53ab7218f152449 P 2a5a1853f59b4b63a2f6cd0c618f97f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:44:17.710189 2022 raft_consensus.cc:3037] T e3cf9e5729dd4ee0b53ab7218f152449 P 2a5a1853f59b4b63a2f6cd0c618f97f1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:44:17.711216 2022 raft_consensus.cc:504] T e3cf9e5729dd4ee0b53ab7218f152449 P 2a5a1853f59b4b63a2f6cd0c618f97f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a5a1853f59b4b63a2f6cd0c618f97f1" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43321 } } peers { permanent_uuid: "75fd7b5a529546e4b2428c240711a58b" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37773 } } peers { permanent_uuid: "2f218b5c8db54a2ba8ecddc69387a36a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 46817 } }
I20260925 04:44:17.711630 2022 leader_election.cc:288] T e3cf9e5729dd4ee0b53ab7218f152449 P 2a5a1853f59b4b63a2f6cd0c618f97f1 [CANDIDATE]: Term 1 election: Requested vote from peers 75fd7b5a529546e4b2428c240711a58b (127.0.98.66:37773), 2f218b5c8db54a2ba8ecddc69387a36a (127.0.98.67:46817)
I20260925 04:44:17.711979 1805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3cf9e5729dd4ee0b53ab7218f152449" candidate_uuid: "2a5a1853f59b4b63a2f6cd0c618f97f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75fd7b5a529546e4b2428c240711a58b"
I20260925 04:44:17.712126 1805 raft_consensus.cc:3037] T e3cf9e5729dd4ee0b53ab7218f152449 P 75fd7b5a529546e4b2428c240711a58b [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:44:17.712085 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3cf9e5729dd4ee0b53ab7218f152449" candidate_uuid: "2a5a1853f59b4b63a2f6cd0c618f97f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f218b5c8db54a2ba8ecddc69387a36a"
I20260925 04:44:17.712236 1945 raft_consensus.cc:3037] T e3cf9e5729dd4ee0b53ab7218f152449 P 2f218b5c8db54a2ba8ecddc69387a36a [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:44:17.713174 1805 raft_consensus.cc:2454] T e3cf9e5729dd4ee0b53ab7218f152449 P 75fd7b5a529546e4b2428c240711a58b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a5a1853f59b4b63a2f6cd0c618f97f1 in term 1.
I20260925 04:44:17.713373 1945 raft_consensus.cc:2454] T e3cf9e5729dd4ee0b53ab7218f152449 P 2f218b5c8db54a2ba8ecddc69387a36a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a5a1853f59b4b63a2f6cd0c618f97f1 in term 1.
I20260925 04:44:17.713475 1599 leader_election.cc:302] T e3cf9e5729dd4ee0b53ab7218f152449 P 2a5a1853f59b4b63a2f6cd0c618f97f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a5a1853f59b4b63a2f6cd0c618f97f1, 75fd7b5a529546e4b2428c240711a58b; no voters:
I20260925 04:44:17.713677 2022 raft_consensus.cc:2781] T e3cf9e5729dd4ee0b53ab7218f152449 P 2a5a1853f59b4b63a2f6cd0c618f97f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:44:17.713900 2022 raft_consensus.cc:686] T e3cf9e5729dd4ee0b53ab7218f152449 P 2a5a1853f59b4b63a2f6cd0c618f97f1 [term 1 LEADER]: Becoming Leader. State: Replica: 2a5a1853f59b4b63a2f6cd0c618f97f1, State: Running, Role: LEADER
I20260925 04:44:17.714206 2022 consensus_queue.cc:237] T e3cf9e5729dd4ee0b53ab7218f152449 P 2a5a1853f59b4b63a2f6cd0c618f97f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a5a1853f59b4b63a2f6cd0c618f97f1" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43321 } } peers { permanent_uuid: "75fd7b5a529546e4b2428c240711a58b" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37773 } } peers { permanent_uuid: "2f218b5c8db54a2ba8ecddc69387a36a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 46817 } }
I20260925 04:44:17.716777 1506 catalog_manager.cc:5495] T e3cf9e5729dd4ee0b53ab7218f152449 P 2a5a1853f59b4b63a2f6cd0c618f97f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a5a1853f59b4b63a2f6cd0c618f97f1 (127.0.98.65). New cstate: current_term: 1 leader_uuid: "2a5a1853f59b4b63a2f6cd0c618f97f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a5a1853f59b4b63a2f6cd0c618f97f1" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "75fd7b5a529546e4b2428c240711a58b" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f218b5c8db54a2ba8ecddc69387a36a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 46817 } health_report { overall_health: UNKNOWN } } }
Sep 25 04:44:17 dist-test-slave-kghz krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311456, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:44:17.769078 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:17.760184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48962 (local address 127.0.98.126:36341)
0925 04:44:17.760334 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:17.760340 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:17.760506 (+ 166us) server_negotiation.cc:407] Connection header received
0925 04:44:17.760673 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:17.760676 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:17.760720 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:17.760807 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:17.761488 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:17.762156 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:17.763120 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:17.763334 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:17.763420 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:17.765823 (+ 2403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:17.765826 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:17.765828 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:17.765848 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:17.767387 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:17.767866 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:17.767869 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:17.767876 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:17.767955 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:17.768225 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:17.768227 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:17.768229 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:17.768395 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:17.768470 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:17.768796 (+ 326us) server_negotiation.cc:299] Negotiation successful
0925 04:44:17.768934 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":67}
I20260925 04:44:17.769094 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:17.760087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:36341 (local address 127.0.0.1:48962)
0925 04:44:17.760412 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:17.760426 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:17.760556 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:17.760851 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:17.760853 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:17.761274 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:17.761371 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:17.761377 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:17.762304 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:17.762307 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:17.762996 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:17.763007 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:17.763459 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:17.763463 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:17.763558 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:17.764233 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:17.764245 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:17.765671 (+ 1426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:44:17.767505 (+ 1834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:17.767514 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:17.767519 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:17.767768 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:17.768051 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:17.768053 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:17.768055 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:17.768142 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:17.768523 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:44:17.768530 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:44:17.768637 (+ 107us) client_negotiation.cc:769] Sending connection context
0925 04:44:17.768740 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 04:44:17.768876 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":137,"threads_started":1}
I20260925 04:44:17.773522 1506 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:48962:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 04:44:17.774413 1506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:44:17.779350 1785 tablet_service.cc:1467] Processing CreateTablet for tablet 41b1ed4509434b88a9f17bae5852ceb4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c78e8688d7294aa9b4d08d55cd6dbbd1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 04:44:17.779356 1645 tablet_service.cc:1467] Processing CreateTablet for tablet 41b1ed4509434b88a9f17bae5852ceb4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c78e8688d7294aa9b4d08d55cd6dbbd1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 04:44:17.779450 1925 tablet_service.cc:1467] Processing CreateTablet for tablet 41b1ed4509434b88a9f17bae5852ceb4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c78e8688d7294aa9b4d08d55cd6dbbd1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 04:44:17.779750 1785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41b1ed4509434b88a9f17bae5852ceb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:44:17.779754 1645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41b1ed4509434b88a9f17bae5852ceb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:44:17.779788 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41b1ed4509434b88a9f17bae5852ceb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:44:17.782328 2016 tablet_bootstrap.cc:492] T 41b1ed4509434b88a9f17bae5852ceb4 P 75fd7b5a529546e4b2428c240711a58b: Bootstrap starting.
I20260925 04:44:17.782899 2015 tablet_bootstrap.cc:492] T 41b1ed4509434b88a9f17bae5852ceb4 P 2a5a1853f59b4b63a2f6cd0c618f97f1: Bootstrap starting.
I20260925 04:44:17.783308 2016 tablet_bootstrap.cc:654] T 41b1ed4509434b88a9f17bae5852ceb4 P 75fd7b5a529546e4b2428c240711a58b: Neither blocks nor log segments found. Creating new log.
I20260925 04:44:17.783903 2015 tablet_bootstrap.cc:654] T 41b1ed4509434b88a9f17bae5852ceb4 P 2a5a1853f59b4b63a2f6cd0c618f97f1: Neither blocks nor log segments found. Creating new log.
I20260925 04:44:17.784993 2014 tablet_bootstrap.cc:492] T 41b1ed4509434b88a9f17bae5852ceb4 P 2f218b5c8db54a2ba8ecddc69387a36a: Bootstrap starting.
I20260925 04:44:17.785027 2016 tablet_bootstrap.cc:492] T 41b1ed4509434b88a9f17bae5852ceb4 P 75fd7b5a529546e4b2428c240711a58b: No bootstrap required, opened a new log
I20260925 04:44:17.785094 2016 ts_tablet_manager.cc:1397] T 41b1ed4509434b88a9f17bae5852ceb4 P 75fd7b5a529546e4b2428c240711a58b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 04:44:17.785554 2016 raft_consensus.cc:348] T 41b1ed4509434b88a9f17bae5852ceb4 P 75fd7b5a529546e4b2428c240711a58b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a5a1853f59b4b63a2f6cd0c618f97f1" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43321 } } peers { permanent_uuid: "75fd7b5a529546e4b2428c240711a58b" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37773 } } peers { permanent_uuid: "2f218b5c8db54a2ba8ecddc69387a36a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 46817 } }
I20260925 04:44:17.785683 2016 raft_consensus.cc:374] T 41b1ed4509434b88a9f17bae5852ceb4 P 75fd7b5a529546e4b2428c240711a58b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:44:17.785720 2016 raft_consensus.cc:729] T 41b1ed4509434b88a9f17bae5852ceb4 P 75fd7b5a529546e4b2428c240711a58b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75fd7b5a529546e4b2428c240711a58b, State: Initialized, Role: FOLLOWER
I20260925 04:44:17.785862 2014 tablet_bootstrap.cc:654] T 41b1ed4509434b88a9f17bae5852ceb4 P 2f218b5c8db54a2ba8ecddc69387a36a: Neither blocks nor log segments found. Creating new log.
I20260925 04:44:17.785892 2016 consensus_queue.cc:260] T 41b1ed4509434b88a9f17bae5852ceb4 P 75fd7b5a529546e4b2428c240711a58b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a5a1853f59b4b63a2f6cd0c618f97f1" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43321 } } peers { permanent_uuid: "75fd7b5a529546e4b2428c240711a58b" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37773 } } peers { permanent_uuid: "2f218b5c8db54a2ba8ecddc69387a36a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 46817 } }
I20260925 04:44:17.786185 2016 ts_tablet_manager.cc:1428] T 41b1ed4509434b88a9f17bae5852ceb4 P 75fd7b5a529546e4b2428c240711a58b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 04:44:17.786177 2021 tablet_replica.cc:439] T 41b1ed4509434b88a9f17bae5852ceb4 P 75fd7b5a529546e4b2428c240711a58b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a5a1853f59b4b63a2f6cd0c618f97f1" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43321 } } peers { permanent_uuid: "75fd7b5a529546e4b2428c240711a58b" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37773 } } peers { permanent_uuid: "2f218b5c8db54a2ba8ecddc69387a36a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 46817 } } }
I20260925 04:44:17.786379 2021 tablet_replica.cc:442] T 41b1ed4509434b88a9f17bae5852ceb4 P 75fd7b5a529546e4b2428c240711a58b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 04:44:17.786768 2015 tablet_bootstrap.cc:492] T 41b1ed4509434b88a9f17bae5852ceb4 P 2a5a1853f59b4b63a2f6cd0c618f97f1: No bootstrap required, opened a new log
I20260925 04:44:17.786865 2015 ts_tablet_manager.cc:1397] T 41b1ed4509434b88a9f17bae5852ceb4 P 2a5a1853f59b4b63a2f6cd0c618f97f1: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
W20260925 04:44:17.786860 1992 tablet.cc:2369] T e3cf9e5729dd4ee0b53ab7218f152449 P 2f218b5c8db54a2ba8ecddc69387a36a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:44:17.786988 2014 tablet_bootstrap.cc:492] T 41b1ed4509434b88a9f17bae5852ceb4 P 2f218b5c8db54a2ba8ecddc69387a36a: No bootstrap required, opened a new log
I20260925 04:44:17.787075 2014 ts_tablet_manager.cc:1397] T 41b1ed4509434b88a9f17bae5852ceb4 P 2f218b5c8db54a2ba8ecddc69387a36a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 04:44:17.787297 2015 raft_consensus.cc:348] T 41b1ed4509434b88a9f17bae5852ceb4 P 2a5a1853f59b4b63a2f6cd0c618f97f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a5a1853f59b4b63a2f6cd0c618f97f1" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43321 } } peers { permanent_uuid: "75fd7b5a529546e4b2428c240711a58b" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37773 } } peers { permanent_uuid: "2f218b5c8db54a2ba8ecddc69387a36a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 46817 } }
I20260925 04:44:17.787432 2015 raft_consensus.cc:374] T 41b1ed4509434b88a9f17bae5852ceb4 P 2a5a1853f59b4b63a2f6cd0c618f97f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:44:17.787465 2015 raft_consensus.cc:729] T 41b1ed4509434b88a9f17bae5852ceb4 P 2a5a1853f59b4b63a2f6cd0c618f97f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a5a1853f59b4b63a2f6cd0c618f97f1, State: Initialized, Role: FOLLOWER
I20260925 04:44:17.787452 2014 raft_consensus.cc:348] T 41b1ed4509434b88a9f17bae5852ceb4 P 2f218b5c8db54a2ba8ecddc69387a36a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a5a1853f59b4b63a2f6cd0c618f97f1" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43321 } } peers { permanent_uuid: "75fd7b5a529546e4b2428c240711a58b" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37773 } } peers { permanent_uuid: "2f218b5c8db54a2ba8ecddc69387a36a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 46817 } }
I20260925 04:44:17.787580 2014 raft_consensus.cc:374] T 41b1ed4509434b88a9f17bae5852ceb4 P 2f218b5c8db54a2ba8ecddc69387a36a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:44:17.787622 2014 raft_consensus.cc:729] T 41b1ed4509434b88a9f17bae5852ceb4 P 2f218b5c8db54a2ba8ecddc69387a36a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f218b5c8db54a2ba8ecddc69387a36a, State: Initialized, Role: FOLLOWER
I20260925 04:44:17.787632 2015 consensus_queue.cc:260] T 41b1ed4509434b88a9f17bae5852ceb4 P 2a5a1853f59b4b63a2f6cd0c618f97f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a5a1853f59b4b63a2f6cd0c618f97f1" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43321 } } peers { permanent_uuid: "75fd7b5a529546e4b2428c240711a58b" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37773 } } peers { permanent_uuid: "2f218b5c8db54a2ba8ecddc69387a36a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 46817 } }
I20260925 04:44:17.787760 2014 consensus_queue.cc:260] T 41b1ed4509434b88a9f17bae5852ceb4 P 2f218b5c8db54a2ba8ecddc69387a36a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a5a1853f59b4b63a2f6cd0c618f97f1" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43321 } } peers { permanent_uuid: "75fd7b5a529546e4b2428c240711a58b" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37773 } } peers { permanent_uuid: "2f218b5c8db54a2ba8ecddc69387a36a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 46817 } }
I20260925 04:44:17.787885 2022 tablet_replica.cc:439] T 41b1ed4509434b88a9f17bae5852ceb4 P 2a5a1853f59b4b63a2f6cd0c618f97f1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a5a1853f59b4b63a2f6cd0c618f97f1" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43321 } } peers { permanent_uuid: "75fd7b5a529546e4b2428c240711a58b" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37773 } } peers { permanent_uuid: "2f218b5c8db54a2ba8ecddc69387a36a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 46817 } } }
I20260925 04:44:17.788028 2022 tablet_replica.cc:442] T 41b1ed4509434b88a9f17bae5852ceb4 P 2a5a1853f59b4b63a2f6cd0c618f97f1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 04:44:17.788041 2014 ts_tablet_manager.cc:1428] T 41b1ed4509434b88a9f17bae5852ceb4 P 2f218b5c8db54a2ba8ecddc69387a36a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 04:44:17.788192 2024 tablet_replica.cc:439] T 41b1ed4509434b88a9f17bae5852ceb4 P 2f218b5c8db54a2ba8ecddc69387a36a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a5a1853f59b4b63a2f6cd0c618f97f1" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43321 } } peers { permanent_uuid: "75fd7b5a529546e4b2428c240711a58b" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37773 } } peers { permanent_uuid: "2f218b5c8db54a2ba8ecddc69387a36a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 46817 } } }
I20260925 04:44:17.788335 2024 tablet_replica.cc:442] T 41b1ed4509434b88a9f17bae5852ceb4 P 2f218b5c8db54a2ba8ecddc69387a36a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 04:44:17.789327 2015 ts_tablet_manager.cc:1428] T 41b1ed4509434b88a9f17bae5852ceb4 P 2a5a1853f59b4b63a2f6cd0c618f97f1: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260925 04:44:17.833348 2021 raft_consensus.cc:482] T 41b1ed4509434b88a9f17bae5852ceb4 P 75fd7b5a529546e4b2428c240711a58b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:44:17.833506 2021 raft_consensus.cc:504] T 41b1ed4509434b88a9f17bae5852ceb4 P 75fd7b5a529546e4b2428c240711a58b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a5a1853f59b4b63a2f6cd0c618f97f1" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43321 } } peers { permanent_uuid: "75fd7b5a529546e4b2428c240711a58b" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37773 } } peers { permanent_uuid: "2f218b5c8db54a2ba8ecddc69387a36a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 46817 } }
I20260925 04:44:17.834757 2021 leader_election.cc:288] T 41b1ed4509434b88a9f17bae5852ceb4 P 75fd7b5a529546e4b2428c240711a58b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a5a1853f59b4b63a2f6cd0c618f97f1 (127.0.98.65:43321), 2f218b5c8db54a2ba8ecddc69387a36a (127.0.98.67:46817)
I20260925 04:44:17.839812 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:17.835289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:33369 (local address 127.0.98.65:43321)
0925 04:44:17.835396 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:17.835400 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:17.835414 (+ 14us) server_negotiation.cc:407] Connection header received
0925 04:44:17.835538 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:17.835542 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:17.835585 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:17.835735 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:44:17.836089 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:17.836913 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:17.838425 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:17.839074 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:17.839191 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:17.839262 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:17.839605 (+ 343us) server_negotiation.cc:299] Negotiation successful
0925 04:44:17.839679 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 04:44:17.839843 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:17.834962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:43321 (local address 127.0.98.66:33369)
0925 04:44:17.835298 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:17.835314 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:17.835421 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:17.835770 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:17.835775 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:17.835801 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:44:17.835972 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:17.835980 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:17.837078 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:17.837082 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:17.838229 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:17.838239 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:17.839324 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:17.839328 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:17.839479 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:17.839497 (+ 18us) client_negotiation.cc:769] Sending connection context
0925 04:44:17.839587 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 04:44:17.839679 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":68,"thread_start_us":80,"threads_started":1}
I20260925 04:44:17.839848 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:17.835115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:35247 (local address 127.0.98.67:46817)
0925 04:44:17.835242 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:17.835247 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:17.835457 (+ 210us) server_negotiation.cc:407] Connection header received
0925 04:44:17.835656 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:17.835662 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:17.835727 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:17.835818 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:44:17.836195 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:17.836960 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:17.838589 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:17.839220 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:17.839313 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:17.839366 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:17.839647 (+ 281us) server_negotiation.cc:299] Negotiation successful
0925 04:44:17.839728 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 04:44:17.840173 2056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:17.834988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:46817 (local address 127.0.98.66:35247)
0925 04:44:17.835388 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:17.835400 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:17.835527 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:17.835904 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:17.835907 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:17.835927 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:44:17.836039 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:17.836045 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:17.837084 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:17.837088 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:17.838455 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:17.838466 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:17.839342 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:17.839346 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:17.839508 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:17.839533 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 04:44:17.839972 (+ 439us) client_negotiation.cc:240] Negotiation successful
0925 04:44:17.840041 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"mutex_wait_us":24,"thread_start_us":65,"threads_started":1}
I20260925 04:44:17.840705 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41b1ed4509434b88a9f17bae5852ceb4" candidate_uuid: "75fd7b5a529546e4b2428c240711a58b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a5a1853f59b4b63a2f6cd0c618f97f1" is_pre_election: true
I20260925 04:44:17.840787 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41b1ed4509434b88a9f17bae5852ceb4" candidate_uuid: "75fd7b5a529546e4b2428c240711a58b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f218b5c8db54a2ba8ecddc69387a36a" is_pre_election: true
I20260925 04:44:17.840878 1665 raft_consensus.cc:2454] T 41b1ed4509434b88a9f17bae5852ceb4 P 2a5a1853f59b4b63a2f6cd0c618f97f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75fd7b5a529546e4b2428c240711a58b in term 0.
I20260925 04:44:17.840924 1945 raft_consensus.cc:2454] T 41b1ed4509434b88a9f17bae5852ceb4 P 2f218b5c8db54a2ba8ecddc69387a36a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75fd7b5a529546e4b2428c240711a58b in term 0.
I20260925 04:44:17.841234 1739 leader_election.cc:302] T 41b1ed4509434b88a9f17bae5852ceb4 P 75fd7b5a529546e4b2428c240711a58b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a5a1853f59b4b63a2f6cd0c618f97f1, 75fd7b5a529546e4b2428c240711a58b; no voters:
I20260925 04:44:17.841487 2021 raft_consensus.cc:2781] T 41b1ed4509434b88a9f17bae5852ceb4 P 75fd7b5a529546e4b2428c240711a58b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:44:17.841583 2021 raft_consensus.cc:482] T 41b1ed4509434b88a9f17bae5852ceb4 P 75fd7b5a529546e4b2428c240711a58b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:44:17.841622 2021 raft_consensus.cc:3037] T 41b1ed4509434b88a9f17bae5852ceb4 P 75fd7b5a529546e4b2428c240711a58b [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:44:17.842877 2021 raft_consensus.cc:504] T 41b1ed4509434b88a9f17bae5852ceb4 P 75fd7b5a529546e4b2428c240711a58b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a5a1853f59b4b63a2f6cd0c618f97f1" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43321 } } peers { permanent_uuid: "75fd7b5a529546e4b2428c240711a58b" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37773 } } peers { permanent_uuid: "2f218b5c8db54a2ba8ecddc69387a36a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 46817 } }
I20260925 04:44:17.843256 2021 leader_election.cc:288] T 41b1ed4509434b88a9f17bae5852ceb4 P 75fd7b5a529546e4b2428c240711a58b [CANDIDATE]: Term 1 election: Requested vote from peers 2a5a1853f59b4b63a2f6cd0c618f97f1 (127.0.98.65:43321), 2f218b5c8db54a2ba8ecddc69387a36a (127.0.98.67:46817)
I20260925 04:44:17.843600 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41b1ed4509434b88a9f17bae5852ceb4" candidate_uuid: "75fd7b5a529546e4b2428c240711a58b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f218b5c8db54a2ba8ecddc69387a36a"
I20260925 04:44:17.843587 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41b1ed4509434b88a9f17bae5852ceb4" candidate_uuid: "75fd7b5a529546e4b2428c240711a58b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a5a1853f59b4b63a2f6cd0c618f97f1"
I20260925 04:44:17.843716 1945 raft_consensus.cc:3037] T 41b1ed4509434b88a9f17bae5852ceb4 P 2f218b5c8db54a2ba8ecddc69387a36a [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:44:17.843737 1665 raft_consensus.cc:3037] T 41b1ed4509434b88a9f17bae5852ceb4 P 2a5a1853f59b4b63a2f6cd0c618f97f1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:44:17.844679 1665 raft_consensus.cc:2454] T 41b1ed4509434b88a9f17bae5852ceb4 P 2a5a1853f59b4b63a2f6cd0c618f97f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75fd7b5a529546e4b2428c240711a58b in term 1.
I20260925 04:44:17.844794 1945 raft_consensus.cc:2454] T 41b1ed4509434b88a9f17bae5852ceb4 P 2f218b5c8db54a2ba8ecddc69387a36a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75fd7b5a529546e4b2428c240711a58b in term 1.
I20260925 04:44:17.844957 1739 leader_election.cc:302] T 41b1ed4509434b88a9f17bae5852ceb4 P 75fd7b5a529546e4b2428c240711a58b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a5a1853f59b4b63a2f6cd0c618f97f1, 75fd7b5a529546e4b2428c240711a58b; no voters:
I20260925 04:44:17.845177 2021 raft_consensus.cc:2781] T 41b1ed4509434b88a9f17bae5852ceb4 P 75fd7b5a529546e4b2428c240711a58b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:44:17.845386 2021 raft_consensus.cc:686] T 41b1ed4509434b88a9f17bae5852ceb4 P 75fd7b5a529546e4b2428c240711a58b [term 1 LEADER]: Becoming Leader. State: Replica: 75fd7b5a529546e4b2428c240711a58b, State: Running, Role: LEADER
I20260925 04:44:17.845644 2021 consensus_queue.cc:237] T 41b1ed4509434b88a9f17bae5852ceb4 P 75fd7b5a529546e4b2428c240711a58b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a5a1853f59b4b63a2f6cd0c618f97f1" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43321 } } peers { permanent_uuid: "75fd7b5a529546e4b2428c240711a58b" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37773 } } peers { permanent_uuid: "2f218b5c8db54a2ba8ecddc69387a36a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 46817 } }
I20260925 04:44:17.846699 2021 tablet_replica.cc:439] T 41b1ed4509434b88a9f17bae5852ceb4 P 75fd7b5a529546e4b2428c240711a58b: TxnStatusTablet state changed. Reason: New leader 75fd7b5a529546e4b2428c240711a58b. Latest consensus state: current_term: 1 leader_uuid: "75fd7b5a529546e4b2428c240711a58b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a5a1853f59b4b63a2f6cd0c618f97f1" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43321 } } peers { permanent_uuid: "75fd7b5a529546e4b2428c240711a58b" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37773 } } peers { permanent_uuid: "2f218b5c8db54a2ba8ecddc69387a36a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 46817 } } }
I20260925 04:44:17.846791 2021 tablet_replica.cc:442] T 41b1ed4509434b88a9f17bae5852ceb4 P 75fd7b5a529546e4b2428c240711a58b: This TxnStatusTablet replica's current role is: LEADER
I20260925 04:44:17.847388 2060 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 04:44:17.847551 2060 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 04:44:17.848063 1508 catalog_manager.cc:5495] T 41b1ed4509434b88a9f17bae5852ceb4 P 75fd7b5a529546e4b2428c240711a58b reported cstate change: term changed from 0 to 1, leader changed from <none> to 75fd7b5a529546e4b2428c240711a58b (127.0.98.66). New cstate: current_term: 1 leader_uuid: "75fd7b5a529546e4b2428c240711a58b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a5a1853f59b4b63a2f6cd0c618f97f1" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75fd7b5a529546e4b2428c240711a58b" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f218b5c8db54a2ba8ecddc69387a36a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 46817 } health_report { overall_health: UNKNOWN } } }
I20260925 04:44:17.867944 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:17.863889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:37773 (local address 127.0.0.1:43044)
0925 04:44:17.864021 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:17.864039 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:17.864145 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:17.864474 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:17.864476 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:17.864483 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 04:44:17.864577 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:17.864581 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:17.865577 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:17.865579 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:17.866188 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:17.866196 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:17.866780 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:17.866782 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:17.866848 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:17.866890 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 04:44:17.867668 (+ 778us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 04:44:17.867682 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 04:44:17.867752 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 04:44:17.867829 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260925 04:44:17.868014 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:17.863971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43044 (local address 127.0.98.66:37773)
0925 04:44:17.864102 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:17.864106 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:17.864121 (+ 15us) server_negotiation.cc:407] Connection header received
0925 04:44:17.864287 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:17.864290 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:17.864353 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:17.864461 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:44:17.864674 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:17.865465 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:17.866333 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:17.866675 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:17.866776 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:17.867027 (+ 251us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:44:17.867560 (+ 533us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:44:17.867678 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:17.867777 (+ 99us) server_negotiation.cc:299] Negotiation successful
0925 04:44:17.867832 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 04:44:17.874010 1945 raft_consensus.cc:1261] T 41b1ed4509434b88a9f17bae5852ceb4 P 2f218b5c8db54a2ba8ecddc69387a36a [term 1 FOLLOWER]: Refusing update from remote peer 75fd7b5a529546e4b2428c240711a58b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:44:17.874635 1665 raft_consensus.cc:1261] T 41b1ed4509434b88a9f17bae5852ceb4 P 2a5a1853f59b4b63a2f6cd0c618f97f1 [term 1 FOLLOWER]: Refusing update from remote peer 75fd7b5a529546e4b2428c240711a58b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:44:17.874709 2021 consensus_queue.cc:1035] T 41b1ed4509434b88a9f17bae5852ceb4 P 75fd7b5a529546e4b2428c240711a58b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f218b5c8db54a2ba8ecddc69387a36a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:44:17.876727 2021 consensus_queue.cc:1035] T 41b1ed4509434b88a9f17bae5852ceb4 P 75fd7b5a529546e4b2428c240711a58b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a5a1853f59b4b63a2f6cd0c618f97f1" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:44:17.877277 2024 tablet_replica.cc:439] T 41b1ed4509434b88a9f17bae5852ceb4 P 2f218b5c8db54a2ba8ecddc69387a36a: TxnStatusTablet state changed. Reason: New leader 75fd7b5a529546e4b2428c240711a58b. Latest consensus state: current_term: 1 leader_uuid: "75fd7b5a529546e4b2428c240711a58b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a5a1853f59b4b63a2f6cd0c618f97f1" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43321 } } peers { permanent_uuid: "75fd7b5a529546e4b2428c240711a58b" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37773 } } peers { permanent_uuid: "2f218b5c8db54a2ba8ecddc69387a36a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 46817 } } }
I20260925 04:44:17.877401 2024 tablet_replica.cc:442] T 41b1ed4509434b88a9f17bae5852ceb4 P 2f218b5c8db54a2ba8ecddc69387a36a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 04:44:17.878289 2022 tablet_replica.cc:439] T 41b1ed4509434b88a9f17bae5852ceb4 P 2a5a1853f59b4b63a2f6cd0c618f97f1: TxnStatusTablet state changed. Reason: New leader 75fd7b5a529546e4b2428c240711a58b. Latest consensus state: current_term: 1 leader_uuid: "75fd7b5a529546e4b2428c240711a58b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a5a1853f59b4b63a2f6cd0c618f97f1" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43321 } } peers { permanent_uuid: "75fd7b5a529546e4b2428c240711a58b" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37773 } } peers { permanent_uuid: "2f218b5c8db54a2ba8ecddc69387a36a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 46817 } } }
I20260925 04:44:17.878418 2022 tablet_replica.cc:442] T 41b1ed4509434b88a9f17bae5852ceb4 P 2a5a1853f59b4b63a2f6cd0c618f97f1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 04:44:17.879174 2021 tablet_replica.cc:439] T 41b1ed4509434b88a9f17bae5852ceb4 P 75fd7b5a529546e4b2428c240711a58b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "75fd7b5a529546e4b2428c240711a58b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a5a1853f59b4b63a2f6cd0c618f97f1" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43321 } } peers { permanent_uuid: "75fd7b5a529546e4b2428c240711a58b" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37773 } } peers { permanent_uuid: "2f218b5c8db54a2ba8ecddc69387a36a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 46817 } } }
I20260925 04:44:17.879282 2021 tablet_replica.cc:442] T 41b1ed4509434b88a9f17bae5852ceb4 P 75fd7b5a529546e4b2428c240711a58b: This TxnStatusTablet replica's current role is: LEADER
I20260925 04:44:17.880499 2024 tablet_replica.cc:439] T 41b1ed4509434b88a9f17bae5852ceb4 P 2f218b5c8db54a2ba8ecddc69387a36a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "75fd7b5a529546e4b2428c240711a58b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a5a1853f59b4b63a2f6cd0c618f97f1" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43321 } } peers { permanent_uuid: "75fd7b5a529546e4b2428c240711a58b" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37773 } } peers { permanent_uuid: "2f218b5c8db54a2ba8ecddc69387a36a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 46817 } } }
I20260925 04:44:17.880614 2024 tablet_replica.cc:442] T 41b1ed4509434b88a9f17bae5852ceb4 P 2f218b5c8db54a2ba8ecddc69387a36a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 04:44:17.881238 2027 mvcc.cc:205] Tried to move back new op lower bound from 7333115731444867072 to 7333115731338563584. Current Snapshot: MvccSnapshot[applied={T|T < 7333115731444867072 or (T in {7333115731444867072})}]
I20260925 04:44:17.882863 2058 tablet_replica.cc:439] T 41b1ed4509434b88a9f17bae5852ceb4 P 75fd7b5a529546e4b2428c240711a58b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "75fd7b5a529546e4b2428c240711a58b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a5a1853f59b4b63a2f6cd0c618f97f1" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43321 } } peers { permanent_uuid: "75fd7b5a529546e4b2428c240711a58b" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37773 } } peers { permanent_uuid: "2f218b5c8db54a2ba8ecddc69387a36a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 46817 } } }
I20260925 04:44:17.883004 2058 tablet_replica.cc:442] T 41b1ed4509434b88a9f17bae5852ceb4 P 75fd7b5a529546e4b2428c240711a58b: This TxnStatusTablet replica's current role is: LEADER
I20260925 04:44:17.883581 2022 tablet_replica.cc:439] T 41b1ed4509434b88a9f17bae5852ceb4 P 2a5a1853f59b4b63a2f6cd0c618f97f1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "75fd7b5a529546e4b2428c240711a58b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a5a1853f59b4b63a2f6cd0c618f97f1" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43321 } } peers { permanent_uuid: "75fd7b5a529546e4b2428c240711a58b" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37773 } } peers { permanent_uuid: "2f218b5c8db54a2ba8ecddc69387a36a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 46817 } } }
I20260925 04:44:17.883697 2022 tablet_replica.cc:442] T 41b1ed4509434b88a9f17bae5852ceb4 P 2a5a1853f59b4b63a2f6cd0c618f97f1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 04:44:17.889686 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:17.885796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59338 (local address 127.0.98.65:43321)
0925 04:44:17.885902 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:17.885906 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:17.885943 (+ 37us) server_negotiation.cc:407] Connection header received
0925 04:44:17.886129 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:17.886131 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:17.886180 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:17.886320 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:44:17.886523 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:17.887007 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:17.888130 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:17.888370 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:17.888458 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:17.888740 (+ 282us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:44:17.889293 (+ 553us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:44:17.889392 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:17.889472 (+ 80us) server_negotiation.cc:299] Negotiation successful
0925 04:44:17.889523 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 04:44:17.900699 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:17.896752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:56739 (local address 127.0.98.66:37773)
0925 04:44:17.896860 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:17.896865 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:17.897167 (+ 302us) server_negotiation.cc:407] Connection header received
0925 04:44:17.897272 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:17.897274 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:17.897319 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:17.897386 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:44:17.897646 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:17.898290 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:17.899574 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:17.899848 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:17.899931 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:17.900267 (+ 336us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:44:17.900339 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:44:17.900398 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:17.900544 (+ 146us) server_negotiation.cc:299] Negotiation successful
0925 04:44:17.900593 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 04:44:17.900750 2068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:17.896603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:37773 (local address 127.0.98.65:56739)
0925 04:44:17.897053 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:17.897067 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:17.897175 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:17.897412 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:17.897415 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:17.897423 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 04:44:17.897542 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:17.897548 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:17.898389 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:17.898392 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:17.899416 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:17.899426 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:17.899955 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:17.899959 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:17.900061 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:17.900113 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 04:44:17.900422 (+ 309us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 04:44:17.900428 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 04:44:17.900495 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 04:44:17.900572 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":135,"threads_started":1}
I20260925 04:44:17.907652 1945 raft_consensus.cc:1261] T e3cf9e5729dd4ee0b53ab7218f152449 P 2f218b5c8db54a2ba8ecddc69387a36a [term 1 FOLLOWER]: Refusing update from remote peer 2a5a1853f59b4b63a2f6cd0c618f97f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:44:17.907671 1805 raft_consensus.cc:1261] T e3cf9e5729dd4ee0b53ab7218f152449 P 75fd7b5a529546e4b2428c240711a58b [term 1 FOLLOWER]: Refusing update from remote peer 2a5a1853f59b4b63a2f6cd0c618f97f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:44:17.908139 2045 consensus_queue.cc:1035] T e3cf9e5729dd4ee0b53ab7218f152449 P 2a5a1853f59b4b63a2f6cd0c618f97f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f218b5c8db54a2ba8ecddc69387a36a" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:44:17.908365 2022 consensus_queue.cc:1035] T e3cf9e5729dd4ee0b53ab7218f152449 P 2a5a1853f59b4b63a2f6cd0c618f97f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75fd7b5a529546e4b2428c240711a58b" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:44:17.932452 2072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:17.928622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:43321 (local address 127.0.98.66:45053)
0925 04:44:17.929013 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:17.929029 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:17.929135 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:17.929348 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:17.929351 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:17.929363 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 04:44:17.929496 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:17.929502 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:17.930310 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:17.930317 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:17.931261 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:17.931274 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:17.931806 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:17.931808 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:17.931878 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:17.931921 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 04:44:17.932207 (+ 286us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 04:44:17.932212 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 04:44:17.932261 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 04:44:17.932325 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":132,"threads_started":1}
I20260925 04:44:17.932456 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:17.928726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:45053 (local address 127.0.98.65:43321)
0925 04:44:17.928844 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:17.928849 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:17.929119 (+ 270us) server_negotiation.cc:407] Connection header received
0925 04:44:17.929214 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:17.929216 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:17.929257 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:17.929319 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:44:17.929614 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:17.930179 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:17.931367 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:17.931700 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:17.931804 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:17.932050 (+ 246us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:44:17.932129 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:44:17.932198 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:17.932277 (+ 79us) server_negotiation.cc:299] Negotiation successful
0925 04:44:17.932325 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 04:44:17.961057 1507 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48960:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:44:17.961247 1507 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48960:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:44:17.963727 1507 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6f7507ec830a4633868cda169231b809: Sending DeleteTablet for 3 replicas of tablet e3cf9e5729dd4ee0b53ab7218f152449
I20260925 04:44:17.964361 1785 tablet_service.cc:1514] Processing DeleteTablet for tablet e3cf9e5729dd4ee0b53ab7218f152449 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:44:17 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:43042
I20260925 04:44:17.964458 1925 tablet_service.cc:1514] Processing DeleteTablet for tablet e3cf9e5729dd4ee0b53ab7218f152449 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:44:17 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:35794
I20260925 04:44:17.964553 2037 tablet_replica.cc:330] T e3cf9e5729dd4ee0b53ab7218f152449 P 75fd7b5a529546e4b2428c240711a58b: stopping tablet replica
I20260925 04:44:17.964654 2038 tablet_replica.cc:330] T e3cf9e5729dd4ee0b53ab7218f152449 P 2f218b5c8db54a2ba8ecddc69387a36a: stopping tablet replica
I20260925 04:44:17.964682 2037 raft_consensus.cc:2229] T e3cf9e5729dd4ee0b53ab7218f152449 P 75fd7b5a529546e4b2428c240711a58b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:44:17.964771 2038 raft_consensus.cc:2229] T e3cf9e5729dd4ee0b53ab7218f152449 P 2f218b5c8db54a2ba8ecddc69387a36a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:44:17.964815 2037 raft_consensus.cc:2258] T e3cf9e5729dd4ee0b53ab7218f152449 P 75fd7b5a529546e4b2428c240711a58b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:44:17.964890 2038 raft_consensus.cc:2258] T e3cf9e5729dd4ee0b53ab7218f152449 P 2f218b5c8db54a2ba8ecddc69387a36a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:44:17.965404 1645 tablet_service.cc:1514] Processing DeleteTablet for tablet e3cf9e5729dd4ee0b53ab7218f152449 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:44:17 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:59336
I20260925 04:44:17.965584 2039 tablet_replica.cc:330] T e3cf9e5729dd4ee0b53ab7218f152449 P 2a5a1853f59b4b63a2f6cd0c618f97f1: stopping tablet replica
I20260925 04:44:17.965656 2037 ts_tablet_manager.cc:1905] T e3cf9e5729dd4ee0b53ab7218f152449 P 75fd7b5a529546e4b2428c240711a58b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:44:17.965708 2039 raft_consensus.cc:2229] T e3cf9e5729dd4ee0b53ab7218f152449 P 2a5a1853f59b4b63a2f6cd0c618f97f1 [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:44:17.966014 2039 raft_consensus.cc:2258] T e3cf9e5729dd4ee0b53ab7218f152449 P 2a5a1853f59b4b63a2f6cd0c618f97f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:44:17.966151 2038 ts_tablet_manager.cc:1905] T e3cf9e5729dd4ee0b53ab7218f152449 P 2f218b5c8db54a2ba8ecddc69387a36a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:44:17.966795 2039 ts_tablet_manager.cc:1905] T e3cf9e5729dd4ee0b53ab7218f152449 P 2a5a1853f59b4b63a2f6cd0c618f97f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:44:17.968549 2037 ts_tablet_manager.cc:1918] T e3cf9e5729dd4ee0b53ab7218f152449 P 75fd7b5a529546e4b2428c240711a58b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 04:44:17.968641 2037 log.cc:1196] T e3cf9e5729dd4ee0b53ab7218f152449 P 75fd7b5a529546e4b2428c240711a58b: Deleting WAL directory at /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/minicluster-data/ts-1/wal/wals/e3cf9e5729dd4ee0b53ab7218f152449
I20260925 04:44:17.968835 2038 ts_tablet_manager.cc:1918] T e3cf9e5729dd4ee0b53ab7218f152449 P 2f218b5c8db54a2ba8ecddc69387a36a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 04:44:17.968914 2038 log.cc:1196] T e3cf9e5729dd4ee0b53ab7218f152449 P 2f218b5c8db54a2ba8ecddc69387a36a: Deleting WAL directory at /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/minicluster-data/ts-2/wal/wals/e3cf9e5729dd4ee0b53ab7218f152449
I20260925 04:44:17.968936 2037 ts_tablet_manager.cc:1939] T e3cf9e5729dd4ee0b53ab7218f152449 P 75fd7b5a529546e4b2428c240711a58b: Deleting consensus metadata
I20260925 04:44:17.969178 2038 ts_tablet_manager.cc:1939] T e3cf9e5729dd4ee0b53ab7218f152449 P 2f218b5c8db54a2ba8ecddc69387a36a: Deleting consensus metadata
I20260925 04:44:17.969527 2039 ts_tablet_manager.cc:1918] T e3cf9e5729dd4ee0b53ab7218f152449 P 2a5a1853f59b4b63a2f6cd0c618f97f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 04:44:17.969606 2039 log.cc:1196] T e3cf9e5729dd4ee0b53ab7218f152449 P 2a5a1853f59b4b63a2f6cd0c618f97f1: Deleting WAL directory at /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790311449873301-393-0/minicluster-data/ts-0/wal/wals/e3cf9e5729dd4ee0b53ab7218f152449
I20260925 04:44:17.969691 1495 catalog_manager.cc:4841] TS 75fd7b5a529546e4b2428c240711a58b (127.0.98.66:37773): tablet e3cf9e5729dd4ee0b53ab7218f152449 (table test-table [id=df7775d6e90745c0b794d5599ab149da]) successfully deleted
I20260925 04:44:17.969858 2039 ts_tablet_manager.cc:1939] T e3cf9e5729dd4ee0b53ab7218f152449 P 2a5a1853f59b4b63a2f6cd0c618f97f1: Deleting consensus metadata
I20260925 04:44:17.969870 1492 catalog_manager.cc:4841] TS 2f218b5c8db54a2ba8ecddc69387a36a (127.0.98.67:46817): tablet e3cf9e5729dd4ee0b53ab7218f152449 (table test-table [id=df7775d6e90745c0b794d5599ab149da]) successfully deleted
I20260925 04:44:17.970561 1495 catalog_manager.cc:4841] TS 2a5a1853f59b4b63a2f6cd0c618f97f1 (127.0.98.65:43321): tablet e3cf9e5729dd4ee0b53ab7218f152449 (table test-table [id=df7775d6e90745c0b794d5599ab149da]) successfully deleted
Sep 25 04:44:17 dist-test-slave-kghz krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311457, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.65@KRBTEST.COM
I20260925 04:44:17.983871 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:17.971575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59354 (local address 127.0.98.65:43321)
0925 04:44:17.971690 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:17.971695 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:17.972210 (+ 515us) server_negotiation.cc:407] Connection header received
0925 04:44:17.972392 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:17.972395 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:17.972443 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:17.972510 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:17.973465 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:17.974282 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:17.975497 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:17.975841 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:17.975945 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:17.979710 (+ 3765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:17.979722 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:17.979737 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:17.979765 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:17.981853 (+ 2088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:17.982366 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:17.982373 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:17.982375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:17.982465 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:17.982770 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:17.982773 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:17.982775 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:17.983076 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:17.983295 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:17.983481 (+ 186us) server_negotiation.cc:299] Negotiation successful
0925 04:44:17.983622 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260925 04:44:17.984826 1705 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59354
I20260925 04:44:17.996289 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:17.986955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48964 (local address 127.0.98.126:36341)
0925 04:44:17.987097 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:17.987101 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:17.987340 (+ 239us) server_negotiation.cc:407] Connection header received
0925 04:44:17.987450 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:17.987454 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:17.987508 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:17.987601 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:17.988393 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:17.989236 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:17.990439 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:17.990801 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:17.990903 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:17.992639 (+ 1736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:17.992644 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:17.992647 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:17.992676 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:17.994452 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:17.994934 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:17.994938 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:17.994940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:17.995048 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:17.995353 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:17.995356 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:17.995358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:17.995581 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:17.995706 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:17.995989 (+ 283us) server_negotiation.cc:299] Negotiation successful
0925 04:44:17.996112 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260925 04:44:17.996667 1507 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48964
I20260925 04:44:17.998118 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 1580
I20260925 04:44:18.006011 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 1720
I20260925 04:44:18.012959 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 1860
I20260925 04:44:18.019325 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 1476
2026-09-25T04:44:18Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2868 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:44:18 dist-test-slave-kghz krb5kdc[2082](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:44:18 dist-test-slave-kghz krb5kdc[2082](info): set up 2 sockets
Sep 25 04:44:18 dist-test-slave-kghz krb5kdc[2082](info): commencing operation
krb5kdc: starting...
W20260925 04:44:19.114172 393 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/krb5kdc/test-user.keytab.
Sep 25 04:44:19 dist-test-slave-kghz krb5kdc[2082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311459, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:44:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:44:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260925 04:44:19.241556 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:39315
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:46139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:39315
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:44:19.353845 2095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:44:19.354158 2095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:44:19.354203 2095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:44:19.358395 2095 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:44:19.358459 2095 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 04:44:19.358476 2095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:44:19.358494 2095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:44:19.358511 2095 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:44:19.358527 2095 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:44:19.364076 2095 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:46139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:39315
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:39315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2095
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:44:19.365191 2095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:44:19.366191 2095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:44:19.374883 2100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:19.374898 2103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:19.374898 2101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:19.374951 2095 server_base.cc:1034] running on GCE node
I20260925 04:44:19.375804 2095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:44:19.376978 2095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:44:19.378153 2095 hybrid_clock.cc:648] HybridClock initialized: now 1790311459378139 us; error 50 us; skew 500 ppm
Sep 25 04:44:19 dist-test-slave-kghz krb5kdc[2082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311459, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:44:19.381194 2095 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260925 04:44:19.382189 2095 webserver.cc:460] Webserver started at http://127.0.98.126:45549/ using document root <none> and password file <none>
I20260925 04:44:19.382786 2095 fs_manager.cc:359] Metadata directory not provided
I20260925 04:44:19.382850 2095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:44:19.383015 2095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:44:19.385012 2095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/minicluster-data/master-0/data/instance:
uuid: "4821c63c5d2148a69569f787e8953a81"
format_stamp: "Formatted at 2026-09-25 04:44:19 on dist-test-slave-kghz"
server_key: "a19be5b0d5aef457e27b5a10e32359e3"
server_key_iv: "ba2637c66d1a6d9fe09738356cd29319"
server_key_version: "encryptionkey@0"
I20260925 04:44:19.385538 2095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/minicluster-data/master-0/wal/instance:
uuid: "4821c63c5d2148a69569f787e8953a81"
format_stamp: "Formatted at 2026-09-25 04:44:19 on dist-test-slave-kghz"
server_key: "a19be5b0d5aef457e27b5a10e32359e3"
server_key_iv: "ba2637c66d1a6d9fe09738356cd29319"
server_key_version: "encryptionkey@0"
I20260925 04:44:19.389346 2095 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 04:44:19.391983 2110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:19.393374 2095 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 04:44:19.393545 2095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/minicluster-data/master-0/wal
uuid: "4821c63c5d2148a69569f787e8953a81"
format_stamp: "Formatted at 2026-09-25 04:44:19 on dist-test-slave-kghz"
server_key: "a19be5b0d5aef457e27b5a10e32359e3"
server_key_iv: "ba2637c66d1a6d9fe09738356cd29319"
server_key_version: "encryptionkey@0"
I20260925 04:44:19.393676 2095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:44:19.417701 2095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:44:19.421108 2095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:44:19.421343 2095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:44:19.429723 2095 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:39315
I20260925 04:44:19.429744 2172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:39315 every 8 connection(s)
I20260925 04:44:19.430769 2095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
I20260925 04:44:19.433763 2173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:44:19.438464 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 2095
I20260925 04:44:19.438586 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/minicluster-data/master-0/wal/instance
I20260925 04:44:19.438855 393 external_mini_cluster.cc:1428] Setting key 8bb1cf9aff84de7dc851703ac90973c9
I20260925 04:44:19.440407 2173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4821c63c5d2148a69569f787e8953a81: Bootstrap starting.
I20260925 04:44:19.442929 2173 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4821c63c5d2148a69569f787e8953a81: Neither blocks nor log segments found. Creating new log.
I20260925 04:44:19.443722 2173 log.cc:824] T 00000000000000000000000000000000 P 4821c63c5d2148a69569f787e8953a81: Log is configured to *not* fsync() on all Append() calls
I20260925 04:44:19.445685 2173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4821c63c5d2148a69569f787e8953a81: No bootstrap required, opened a new log
Sep 25 04:44:19 dist-test-slave-kghz krb5kdc[2082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311459, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:44:19.449671 2173 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4821c63c5d2148a69569f787e8953a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4821c63c5d2148a69569f787e8953a81" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39315 } }
I20260925 04:44:19.449949 2173 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4821c63c5d2148a69569f787e8953a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:44:19.450039 2173 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4821c63c5d2148a69569f787e8953a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4821c63c5d2148a69569f787e8953a81, State: Initialized, Role: FOLLOWER
I20260925 04:44:19.450568 2173 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4821c63c5d2148a69569f787e8953a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4821c63c5d2148a69569f787e8953a81" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39315 } }
I20260925 04:44:19.450726 2173 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4821c63c5d2148a69569f787e8953a81 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:44:19.450794 2173 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4821c63c5d2148a69569f787e8953a81 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:44:19.450898 2173 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4821c63c5d2148a69569f787e8953a81 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:44:19.451797 2173 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4821c63c5d2148a69569f787e8953a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4821c63c5d2148a69569f787e8953a81" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39315 } }
I20260925 04:44:19.452149 2173 leader_election.cc:302] T 00000000000000000000000000000000 P 4821c63c5d2148a69569f787e8953a81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4821c63c5d2148a69569f787e8953a81; no voters:
I20260925 04:44:19.452497 2173 leader_election.cc:288] T 00000000000000000000000000000000 P 4821c63c5d2148a69569f787e8953a81 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:44:19.452644 2178 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4821c63c5d2148a69569f787e8953a81 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:44:19.452878 2178 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4821c63c5d2148a69569f787e8953a81 [term 1 LEADER]: Becoming Leader. State: Replica: 4821c63c5d2148a69569f787e8953a81, State: Running, Role: LEADER
I20260925 04:44:19.453213 2178 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4821c63c5d2148a69569f787e8953a81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4821c63c5d2148a69569f787e8953a81" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39315 } }
I20260925 04:44:19.453802 2176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:19.440437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59742 (local address 127.0.98.126:39315)
0925 04:44:19.440936 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:19.440952 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:19.441012 (+ 60us) server_negotiation.cc:407] Connection header received
0925 04:44:19.441737 (+ 725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:19.441754 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:19.442120 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:19.442547 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:19.443397 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:19.444477 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:19.445401 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:19.445940 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:19.446118 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:19.448627 (+ 2509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:19.448632 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:19.448647 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:19.448674 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:19.450586 (+ 1912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:19.451162 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:19.451177 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:19.451182 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:19.451292 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:19.451593 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:19.451596 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:19.451598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:19.452060 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:19.452260 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:19.452523 (+ 263us) server_negotiation.cc:299] Negotiation successful
0925 04:44:19.452855 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":149,"threads_started":1}
I20260925 04:44:19.453851 2173 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4821c63c5d2148a69569f787e8953a81 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:44:19.455117 2179 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4821c63c5d2148a69569f787e8953a81 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4821c63c5d2148a69569f787e8953a81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4821c63c5d2148a69569f787e8953a81" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39315 } } }
I20260925 04:44:19.455246 2179 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4821c63c5d2148a69569f787e8953a81 [sys.catalog]: This master's current role is: LEADER
I20260925 04:44:19.455536 2180 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4821c63c5d2148a69569f787e8953a81 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4821c63c5d2148a69569f787e8953a81. Latest consensus state: current_term: 1 leader_uuid: "4821c63c5d2148a69569f787e8953a81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4821c63c5d2148a69569f787e8953a81" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39315 } } }
I20260925 04:44:19.455621 2180 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4821c63c5d2148a69569f787e8953a81 [sys.catalog]: This master's current role is: LEADER
I20260925 04:44:19.456545 2187 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:44:19.459164 2187 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:44:19.465123 2187 catalog_manager.cc:1347] Generated new cluster ID: e8564854b3514862bb1501c6a5a9fd57
I20260925 04:44:19.465205 2187 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:44:19.485425 2187 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:44:19.486668 2187 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:44:19.498081 2187 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4821c63c5d2148a69569f787e8953a81: Generated new TSK 0
I20260925 04:44:19.498950 2187 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/krb5kdc/kudu_127.0.98.65.keytab.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/krb5kdc/kudu_127.0.98.65.keytab.
I20260925 04:44:19.537433 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:0
--local_ip_for_outbound_sockets=127.0.98.65
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:39315
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:46139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:44:19.655865 2199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:44:19.656157 2199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:44:19.656206 2199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:44:19.660537 2199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:44:19.660612 2199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:44:19.660655 2199 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 04:44:19.660669 2199 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 04:44:19.660702 2199 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260925 04:44:19.666213 2199 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:46139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.126:39315
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:44:19.667287 2199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:44:19.668285 2199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:44:19.678947 2204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:19.679100 2205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:19.679049 2207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:19.679186 2199 server_base.cc:1034] running on GCE node
I20260925 04:44:19.679661 2199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:44:19.680352 2199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:44:19.681537 2199 hybrid_clock.cc:648] HybridClock initialized: now 1790311459681513 us; error 39 us; skew 500 ppm
Sep 25 04:44:19 dist-test-slave-kghz krb5kdc[2082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311459, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:44:19.684614 2199 init.cc:377] Logged in from keytab as kudu/127.0.98.65@KRBTEST.COM (short username kudu)
I20260925 04:44:19.685600 2199 webserver.cc:460] Webserver started at http://127.0.98.65:43319/ using document root <none> and password file <none>
I20260925 04:44:19.686199 2199 fs_manager.cc:359] Metadata directory not provided
I20260925 04:44:19.686340 2199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:44:19.686599 2199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:44:19.688748 2199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/minicluster-data/ts-0/data/instance:
uuid: "926c2275bff542c48e377eba68599697"
format_stamp: "Formatted at 2026-09-25 04:44:19 on dist-test-slave-kghz"
server_key: "78c82a9ad5cd0806d201bd03e31dfa58"
server_key_iv: "5030fbf06a1ab779b926e9b4a0d99caf"
server_key_version: "encryptionkey@0"
I20260925 04:44:19.689329 2199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/minicluster-data/ts-0/wal/instance:
uuid: "926c2275bff542c48e377eba68599697"
format_stamp: "Formatted at 2026-09-25 04:44:19 on dist-test-slave-kghz"
server_key: "78c82a9ad5cd0806d201bd03e31dfa58"
server_key_iv: "5030fbf06a1ab779b926e9b4a0d99caf"
server_key_version: "encryptionkey@0"
I20260925 04:44:19.693313 2199 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260925 04:44:19.695881 2214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:19.697365 2199 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 04:44:19.697522 2199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/minicluster-data/ts-0/wal
uuid: "926c2275bff542c48e377eba68599697"
format_stamp: "Formatted at 2026-09-25 04:44:19 on dist-test-slave-kghz"
server_key: "78c82a9ad5cd0806d201bd03e31dfa58"
server_key_iv: "5030fbf06a1ab779b926e9b4a0d99caf"
server_key_version: "encryptionkey@0"
I20260925 04:44:19.697649 2199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:44:19.719349 2199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:44:19.722833 2199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:44:19.723059 2199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:44:19.724594 2199 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:44:19.724644 2199 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:19.724709 2199 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:44:19.724740 2199 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 04:44:19 dist-test-slave-kghz krb5kdc[2082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311459, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:44:19.735558 2199 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:46207
I20260925 04:44:19.735594 2329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:46207 every 8 connection(s)
I20260925 04:44:19.736672 2199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
I20260925 04:44:19.739055 2176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:19.725618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:44969 (local address 127.0.98.126:39315)
0925 04:44:19.725764 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:19.725769 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:19.726680 (+ 911us) server_negotiation.cc:407] Connection header received
0925 04:44:19.727738 (+ 1058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:19.727742 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:19.727801 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:19.727915 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:19.729622 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:19.730348 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:19.731548 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:19.731843 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:19.731965 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:19.735393 (+ 3428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:19.735397 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:19.735400 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:19.735432 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:19.737007 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:19.737612 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:19.737616 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:19.737618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:19.737709 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:19.738024 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:19.738026 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:19.738027 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:19.738201 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:19.738353 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:19.738815 (+ 462us) server_negotiation.cc:299] Negotiation successful
0925 04:44:19.738928 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 04:44:19.739977 2223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:19.726018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39315 (local address 127.0.98.65:44969)
0925 04:44:19.726514 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:19.726564 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:19.727502 (+ 938us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:19.728152 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:19.728165 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:19.728652 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:19.729419 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:19.729442 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:19.730500 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:19.730504 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:19.731417 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:19.731426 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:19.732023 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:19.732027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:19.732236 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:19.733576 (+ 1340us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:19.733599 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:19.735173 (+ 1574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:44:19.737129 (+ 1956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:19.737135 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:19.737152 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:19.737503 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:19.737800 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:19.737802 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:19.737803 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:19.737932 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:19.738344 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:44:19.738354 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:44:19.738588 (+ 234us) client_negotiation.cc:769] Sending connection context
0925 04:44:19.738756 (+ 168us) client_negotiation.cc:240] Negotiation successful
0925 04:44:19.739097 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":133,"threads_started":1}
I20260925 04:44:19.741285 2330 heartbeater.cc:347] Connected to a master server at 127.0.98.126:39315
I20260925 04:44:19.741447 2330 heartbeater.cc:464] Registering TS with master...
I20260925 04:44:19.741969 2330 heartbeater.cc:511] Master 127.0.98.126:39315 requested a full tablet report, sending...
I20260925 04:44:19.743738 2126 ts_manager.cc:194] Registered new tserver with Master: 926c2275bff542c48e377eba68599697 (127.0.98.65:46207)
I20260925 04:44:19.745062 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 2199
I20260925 04:44:19.745167 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/minicluster-data/ts-0/wal/instance
I20260925 04:44:19.745311 2126 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.65@KRBTEST.COM'} at 127.0.98.65:44969
I20260925 04:44:19.745478 393 external_mini_cluster.cc:1428] Setting key 52e200b0ffe7222cf82b9729c937d072
I20260925 04:44:19.752084 2336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:19.743576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39315 (local address 127.0.98.65:58673)
0925 04:44:19.743880 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:19.743896 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:19.744008 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:19.744240 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:19.744243 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:19.744455 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:19.744575 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:19.744581 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:19.745456 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:19.745461 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:19.746234 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:19.746267 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:19.746815 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:19.746819 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:19.746966 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:19.747550 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:19.747568 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:19.748073 (+ 505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:44:19.750323 (+ 2250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:19.750327 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:19.750329 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:19.750542 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:19.750878 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:19.750881 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:19.750882 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:19.750971 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:19.751488 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:44:19.751490 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:44:19.751593 (+ 103us) client_negotiation.cc:769] Sending connection context
0925 04:44:19.751746 (+ 153us) client_negotiation.cc:240] Negotiation successful
0925 04:44:19.751884 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":22,"thread_start_us":98,"threads_started":1}
I20260925 04:44:19.752162 2176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:19.743698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:58673 (local address 127.0.98.126:39315)
0925 04:44:19.743821 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:19.743827 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:19.743973 (+ 146us) server_negotiation.cc:407] Connection header received
0925 04:44:19.744095 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:19.744097 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:19.744135 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:19.744199 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:19.744685 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:19.745284 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:19.746373 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:19.746712 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:19.746812 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:19.748231 (+ 1419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:19.748235 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:19.748237 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:19.748274 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:19.750127 (+ 1853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:19.750651 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:19.750658 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:19.750663 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:19.750758 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:19.751101 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:19.751107 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:19.751111 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:19.751371 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:19.751496 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:19.751839 (+ 343us) server_negotiation.cc:299] Negotiation successful
0925 04:44:19.751979 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
WARNING: no policy specified for kudu/127.0.98.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/krb5kdc/kudu_127.0.98.66.keytab.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/krb5kdc/kudu_127.0.98.66.keytab.
I20260925 04:44:19.775789 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:0
--local_ip_for_outbound_sockets=127.0.98.66
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:39315
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:46139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:44:19.897730 2339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:44:19.898036 2339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:44:19.898088 2339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:44:19.902725 2339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:44:19.902801 2339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:44:19.902846 2339 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 04:44:19.902860 2339 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 04:44:19.902938 2339 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260925 04:44:19.909281 2339 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:46139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.126:39315
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2339
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:44:19.910565 2339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:44:19.911713 2339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:44:19.922066 2347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:19.922055 2345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:19.922070 2344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:19.922219 2339 server_base.cc:1034] running on GCE node
I20260925 04:44:19.922787 2339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:44:19.923480 2339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:44:19.924695 2339 hybrid_clock.cc:648] HybridClock initialized: now 1790311459924673 us; error 46 us; skew 500 ppm
Sep 25 04:44:19 dist-test-slave-kghz krb5kdc[2082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311459, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:44:19.928169 2339 init.cc:377] Logged in from keytab as kudu/127.0.98.66@KRBTEST.COM (short username kudu)
I20260925 04:44:19.929062 2339 webserver.cc:460] Webserver started at http://127.0.98.66:38945/ using document root <none> and password file <none>
I20260925 04:44:19.929597 2339 fs_manager.cc:359] Metadata directory not provided
I20260925 04:44:19.929659 2339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:44:19.929826 2339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:44:19.931988 2339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/minicluster-data/ts-1/data/instance:
uuid: "39f2aab90c2a4055830500e2949241a2"
format_stamp: "Formatted at 2026-09-25 04:44:19 on dist-test-slave-kghz"
server_key: "9515ad4f053684c8a0c43e3810d04656"
server_key_iv: "e344afcdbed94e274c6c184305ce202f"
server_key_version: "encryptionkey@0"
I20260925 04:44:19.932683 2339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/minicluster-data/ts-1/wal/instance:
uuid: "39f2aab90c2a4055830500e2949241a2"
format_stamp: "Formatted at 2026-09-25 04:44:19 on dist-test-slave-kghz"
server_key: "9515ad4f053684c8a0c43e3810d04656"
server_key_iv: "e344afcdbed94e274c6c184305ce202f"
server_key_version: "encryptionkey@0"
I20260925 04:44:19.936604 2339 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260925 04:44:19.939150 2354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:19.940461 2339 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 04:44:19.940595 2339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/minicluster-data/ts-1/wal
uuid: "39f2aab90c2a4055830500e2949241a2"
format_stamp: "Formatted at 2026-09-25 04:44:19 on dist-test-slave-kghz"
server_key: "9515ad4f053684c8a0c43e3810d04656"
server_key_iv: "e344afcdbed94e274c6c184305ce202f"
server_key_version: "encryptionkey@0"
I20260925 04:44:19.940708 2339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:44:19.966593 2339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:44:19.970331 2339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:44:19.970551 2339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:44:19.972179 2339 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:44:19.972257 2339 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:19.972328 2339 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:44:19.972376 2339 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 04:44:19 dist-test-slave-kghz krb5kdc[2082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311459, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:44:19.983700 2339 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:46525
I20260925 04:44:19.983721 2469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:46525 every 8 connection(s)
I20260925 04:44:19.984639 2339 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
I20260925 04:44:19.987236 2176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:19.973194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:41955 (local address 127.0.98.126:39315)
0925 04:44:19.973329 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:19.973332 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:19.974420 (+ 1088us) server_negotiation.cc:407] Connection header received
0925 04:44:19.975578 (+ 1158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:19.975582 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:19.975638 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:19.975728 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:19.977379 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:19.978341 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:19.979856 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:19.980092 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:19.980210 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:19.983372 (+ 3162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:19.983378 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:19.983381 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:19.983416 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:19.985068 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:19.985694 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:19.985698 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:19.985700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:19.985802 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:19.986154 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:19.986156 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:19.986157 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:19.986388 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:19.986467 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:19.986964 (+ 497us) server_negotiation.cc:299] Negotiation successful
0925 04:44:19.987116 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 04:44:19.988215 2363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:19.973728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39315 (local address 127.0.98.66:41955)
0925 04:44:19.974212 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:19.974313 (+ 101us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:19.975329 (+ 1016us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:19.975931 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:19.975945 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:19.976477 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:19.977188 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:19.977208 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:19.978500 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:19.978507 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:19.979711 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:19.979722 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:19.980262 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:19.980266 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:19.980452 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:19.981195 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:19.981213 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:19.983113 (+ 1900us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:44:19.985232 (+ 2119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:19.985237 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:19.985253 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:19.985583 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:19.985894 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:19.985896 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:19.985898 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:19.986023 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:19.986481 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:44:19.986487 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:44:19.986751 (+ 264us) client_negotiation.cc:769] Sending connection context
0925 04:44:19.986922 (+ 171us) client_negotiation.cc:240] Negotiation successful
0925 04:44:19.987209 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":125,"threads_started":1}
I20260925 04:44:19.989406 2470 heartbeater.cc:347] Connected to a master server at 127.0.98.126:39315
I20260925 04:44:19.989668 2470 heartbeater.cc:464] Registering TS with master...
I20260925 04:44:19.990188 2470 heartbeater.cc:511] Master 127.0.98.126:39315 requested a full tablet report, sending...
I20260925 04:44:19.991398 2126 ts_manager.cc:194] Registered new tserver with Master: 39f2aab90c2a4055830500e2949241a2 (127.0.98.66:46525)
I20260925 04:44:19.991986 2126 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.66@KRBTEST.COM'} at 127.0.98.66:41955
I20260925 04:44:19.993610 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 2339
I20260925 04:44:19.993737 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/minicluster-data/ts-1/wal/instance
I20260925 04:44:19.994041 393 external_mini_cluster.cc:1428] Setting key bf3f87652f1caee28aee14123afa6c7c
I20260925 04:44:20.001444 2176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:19.992122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:38919 (local address 127.0.98.126:39315)
0925 04:44:19.992236 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:19.992241 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:19.992427 (+ 186us) server_negotiation.cc:407] Connection header received
0925 04:44:19.992540 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:19.992542 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:19.992581 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:19.992649 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:19.994425 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:19.995100 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:19.995959 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:19.996353 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:19.996477 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:19.997755 (+ 1278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:19.997761 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:19.997784 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:19.997816 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:19.999426 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:19.999992 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:19.999996 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:19.999997 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:20.000123 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:20.000464 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:20.000466 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:20.000467 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:20.000714 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:20.000853 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:20.001211 (+ 358us) server_negotiation.cc:299] Negotiation successful
0925 04:44:20.001317 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 04:44:20.001621 2476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:19.992056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39315 (local address 127.0.98.66:38919)
0925 04:44:19.992327 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:19.992342 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:19.992445 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:19.993904 (+ 1459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:19.993906 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:19.994090 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:19.994285 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:19.994294 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:19.995242 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:19.995246 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:19.995835 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:19.995843 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:19.996470 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:19.996472 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:19.996552 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:19.997225 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:19.997246 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:19.997615 (+ 369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:44:19.999576 (+ 1961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:19.999582 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:19.999588 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:19.999877 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:20.000243 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:20.000250 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:20.000254 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:20.000371 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:20.000845 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:44:20.000851 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:44:20.001015 (+ 164us) client_negotiation.cc:769] Sending connection context
0925 04:44:20.001144 (+ 129us) client_negotiation.cc:240] Negotiation successful
0925 04:44:20.001344 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":90,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/krb5kdc/kudu_127.0.98.67.keytab.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/krb5kdc/kudu_127.0.98.67.keytab.
I20260925 04:44:20.026523 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:0
--local_ip_for_outbound_sockets=127.0.98.67
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:39315
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:46139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:44:20.144804 2479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:44:20.145177 2479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:44:20.145287 2479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:44:20.149531 2479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:44:20.149636 2479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:44:20.149710 2479 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 04:44:20.149760 2479 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 04:44:20.149832 2479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260925 04:44:20.155598 2479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:46139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.126:39315
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:44:20.156905 2479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:44:20.158018 2479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:44:20.168018 2485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:20.168182 2479 server_base.cc:1034] running on GCE node
W20260925 04:44:20.168085 2484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:20.168051 2487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:20.168751 2479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:44:20.169454 2479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:44:20.170653 2479 hybrid_clock.cc:648] HybridClock initialized: now 1790311460170636 us; error 42 us; skew 500 ppm
Sep 25 04:44:20 dist-test-slave-kghz krb5kdc[2082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311460, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:44:20.173648 2479 init.cc:377] Logged in from keytab as kudu/127.0.98.67@KRBTEST.COM (short username kudu)
I20260925 04:44:20.174865 2479 webserver.cc:460] Webserver started at http://127.0.98.67:38325/ using document root <none> and password file <none>
I20260925 04:44:20.175496 2479 fs_manager.cc:359] Metadata directory not provided
I20260925 04:44:20.175591 2479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:44:20.175810 2479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:44:20.178081 2479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/minicluster-data/ts-2/data/instance:
uuid: "50fc1fd1972849d4b96b3df8cc2ca598"
format_stamp: "Formatted at 2026-09-25 04:44:20 on dist-test-slave-kghz"
server_key: "00d086384243e36f5d33b49f40d01a71"
server_key_iv: "bc7be1af61f40003c6763417ec2bf1eb"
server_key_version: "encryptionkey@0"
I20260925 04:44:20.178797 2479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/minicluster-data/ts-2/wal/instance:
uuid: "50fc1fd1972849d4b96b3df8cc2ca598"
format_stamp: "Formatted at 2026-09-25 04:44:20 on dist-test-slave-kghz"
server_key: "00d086384243e36f5d33b49f40d01a71"
server_key_iv: "bc7be1af61f40003c6763417ec2bf1eb"
server_key_version: "encryptionkey@0"
I20260925 04:44:20.182946 2479 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 04:44:20.185737 2494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:20.187288 2479 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:44:20.187467 2479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/minicluster-data/ts-2/wal
uuid: "50fc1fd1972849d4b96b3df8cc2ca598"
format_stamp: "Formatted at 2026-09-25 04:44:20 on dist-test-slave-kghz"
server_key: "00d086384243e36f5d33b49f40d01a71"
server_key_iv: "bc7be1af61f40003c6763417ec2bf1eb"
server_key_version: "encryptionkey@0"
I20260925 04:44:20.187604 2479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:44:20.215301 2479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:44:20.219049 2479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:44:20.219277 2479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:44:20.220806 2479 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:44:20.220896 2479 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:20.220970 2479 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:44:20.221015 2479 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 04:44:20 dist-test-slave-kghz krb5kdc[2082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311460, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:44:20.232209 2479 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:38111
I20260925 04:44:20.232228 2609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:38111 every 8 connection(s)
I20260925 04:44:20.233141 2479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
I20260925 04:44:20.233659 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 2479
I20260925 04:44:20.233777 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/minicluster-data/ts-2/wal/instance
I20260925 04:44:20.234140 393 external_mini_cluster.cc:1428] Setting key 2afaac126869c94577199eb56afa305b
I20260925 04:44:20.237243 2176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:20.221803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:50537 (local address 127.0.98.126:39315)
0925 04:44:20.221939 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:20.221942 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:20.223161 (+ 1219us) server_negotiation.cc:407] Connection header received
0925 04:44:20.224384 (+ 1223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:20.224388 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:20.224474 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:20.224576 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:20.226373 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:20.226926 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:20.228961 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:20.229197 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:20.229286 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:20.232576 (+ 3290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:20.232580 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:20.232585 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:20.232617 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:20.234890 (+ 2273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:20.235572 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:20.235575 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:20.235577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:20.235671 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:20.236016 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:20.236020 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:20.236021 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:20.236262 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:20.236382 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:20.236950 (+ 568us) server_negotiation.cc:299] Negotiation successful
0925 04:44:20.237078 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 04:44:20.238160 2503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:20.222431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39315 (local address 127.0.98.67:50537)
0925 04:44:20.222934 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:20.223025 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:20.224108 (+ 1083us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:20.224820 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:20.224833 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:20.225355 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:20.226164 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:20.226190 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:20.227078 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:20.227085 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:20.228801 (+ 1716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:20.228817 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:20.229377 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:20.229382 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:20.229641 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:20.230449 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:20.230475 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:20.232372 (+ 1897us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:44:20.235013 (+ 2641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:20.235019 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:20.235035 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:20.235426 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:20.235770 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:20.235772 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:20.235774 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:20.235911 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:20.236381 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:44:20.236390 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:44:20.236667 (+ 277us) client_negotiation.cc:769] Sending connection context
0925 04:44:20.236859 (+ 192us) client_negotiation.cc:240] Negotiation successful
0925 04:44:20.237207 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":132,"threads_started":1}
I20260925 04:44:20.239492 2610 heartbeater.cc:347] Connected to a master server at 127.0.98.126:39315
I20260925 04:44:20.239686 2610 heartbeater.cc:464] Registering TS with master...
I20260925 04:44:20.240309 2610 heartbeater.cc:511] Master 127.0.98.126:39315 requested a full tablet report, sending...
I20260925 04:44:20.241433 2127 ts_manager.cc:194] Registered new tserver with Master: 50fc1fd1972849d4b96b3df8cc2ca598 (127.0.98.67:38111)
I20260925 04:44:20.242367 2127 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.67@KRBTEST.COM'} at 127.0.98.67:50537
I20260925 04:44:20.248618 393 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 04:44:20.249773 2176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:20.241804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:33843 (local address 127.0.98.126:39315)
0925 04:44:20.241929 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:20.241933 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:20.242204 (+ 271us) server_negotiation.cc:407] Connection header received
0925 04:44:20.242398 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:20.242407 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:20.242450 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:20.242620 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:20.243070 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:20.243576 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:20.244541 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:20.244769 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:20.244874 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:20.245921 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:20.245925 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:20.245927 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:20.245950 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:20.247863 (+ 1913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:20.248418 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:20.248421 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:20.248423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:20.248515 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:20.248869 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:20.248872 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:20.248873 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:20.249061 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:20.249145 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:20.249540 (+ 395us) server_negotiation.cc:299] Negotiation successful
0925 04:44:20.249644 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 04:44:20.249990 2616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:20.241709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39315 (local address 127.0.98.67:33843)
0925 04:44:20.242100 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:20.242115 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:20.242301 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:20.242632 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:20.242635 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:20.242848 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:20.242946 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:20.242952 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:20.243709 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:20.243712 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:20.244425 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:20.244434 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:20.244875 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:20.244879 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:20.244974 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:20.245473 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:20.245487 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:20.245790 (+ 303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:44:20.248068 (+ 2278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:20.248072 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:20.248074 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:20.248319 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:20.248619 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:20.248622 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:20.248624 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:20.248720 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:20.249166 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:44:20.249169 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:44:20.249278 (+ 109us) client_negotiation.cc:769] Sending connection context
0925 04:44:20.249396 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 04:44:20.249571 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":89,"threads_started":1}
Sep 25 04:44:20 dist-test-slave-kghz krb5kdc[2082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311460, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 04:44:20 dist-test-slave-kghz krb5kdc[2082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311460, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:44:20.277307 2176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:20.267157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59746 (local address 127.0.98.126:39315)
0925 04:44:20.267435 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:20.267447 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:20.267470 (+ 23us) server_negotiation.cc:407] Connection header received
0925 04:44:20.267526 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:20.267528 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:20.267569 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:20.267766 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:20.269290 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:20.269924 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:20.271050 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:20.271281 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:20.271401 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:20.273980 (+ 2579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:20.273984 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:20.273986 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:20.274019 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:20.275558 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:20.276026 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:20.276033 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:20.276036 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:20.276128 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:20.276410 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:20.276415 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:20.276417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:20.276613 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:20.276755 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:20.277006 (+ 251us) server_negotiation.cc:299] Negotiation successful
0925 04:44:20.277128 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161}
I20260925 04:44:20.280359 2127 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59746:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:44:20.283502 2127 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:44:20.298943 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:20.292538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55848 (local address 127.0.98.66:46525)
0925 04:44:20.292935 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:20.292947 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:20.293739 (+ 792us) server_negotiation.cc:407] Connection header received
0925 04:44:20.294011 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:20.294015 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:20.294179 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:20.294368 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:44:20.294638 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:20.295783 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:20.297498 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:20.298320 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:20.298435 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:20.298610 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:20.298745 (+ 135us) server_negotiation.cc:299] Negotiation successful
0925 04:44:20.298821 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":125,"threads_started":1}
I20260925 04:44:20.299022 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:20.292410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:46525 (local address 127.0.0.1:55848)
0925 04:44:20.293564 (+ 1154us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:20.293619 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:20.293852 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:20.294353 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:20.294359 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:20.294381 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:44:20.294520 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:20.294527 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:20.295892 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:20.295894 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:20.297316 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:20.297327 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:20.298440 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:20.298444 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:20.298553 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:20.298594 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 04:44:20.298722 (+ 128us) client_negotiation.cc:240] Negotiation successful
0925 04:44:20.298848 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":985,"mutex_wait_us":24,"thread_start_us":83,"threads_started":1}
I20260925 04:44:20.299669 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:20.293159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:46207 (local address 127.0.0.1:49514)
0925 04:44:20.293796 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:20.293811 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:20.293902 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:20.294590 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:20.294593 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:20.294618 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:44:20.294767 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:20.294774 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:20.296034 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:20.296037 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:20.297255 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:20.297263 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:20.299320 (+ 2057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:20.299323 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:20.299417 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:20.299434 (+ 17us) client_negotiation.cc:769] Sending connection context
0925 04:44:20.299509 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 04:44:20.299564 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"mutex_wait_us":89,"spinlock_wait_cycles":11520,"thread_start_us":109,"threads_started":1}
I20260925 04:44:20.300174 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:20.293323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49514 (local address 127.0.98.65:46207)
0925 04:44:20.293689 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:20.293696 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:20.294005 (+ 309us) server_negotiation.cc:407] Connection header received
0925 04:44:20.294085 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:20.294090 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:20.294332 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:20.294460 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:44:20.294892 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:20.295945 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:20.297372 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:20.298320 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:20.298437 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:20.298588 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:20.299668 (+ 1080us) server_negotiation.cc:299] Negotiation successful
0925 04:44:20.299977 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":77,"threads_started":1}
I20260925 04:44:20.300287 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:20.293172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39110 (local address 127.0.98.67:38111)
0925 04:44:20.293567 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:20.293580 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:20.294005 (+ 425us) server_negotiation.cc:407] Connection header received
0925 04:44:20.294085 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:20.294090 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:20.294313 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:20.294460 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:44:20.294902 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:20.295635 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:20.297984 (+ 2349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:20.298857 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:20.298957 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:20.299114 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:20.300030 (+ 916us) server_negotiation.cc:299] Negotiation successful
0925 04:44:20.300131 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":96,"threads_started":1}
I20260925 04:44:20.300525 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:20.292420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:38111 (local address 127.0.0.1:39110)
0925 04:44:20.293772 (+ 1352us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:20.293784 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:20.293902 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:20.294590 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:20.294593 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:20.294619 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:44:20.294785 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:20.294790 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:20.295755 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:20.295760 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:20.297864 (+ 2104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:20.297872 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:20.299176 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:20.299179 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:20.299293 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:20.299318 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 04:44:20.299383 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 04:44:20.299466 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1289,"mutex_wait_us":979,"spinlock_wait_cycles":32000,"thread_start_us":122,"threads_started":1}
I20260925 04:44:20.302285 2404 tablet_service.cc:1467] Processing CreateTablet for tablet 0921b09ac9794d89a007640f14283636 (DEFAULT_TABLE table=test-table [id=caf3d629ba2a4bab86e8fbbb6d91b52b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:44:20.302920 2544 tablet_service.cc:1467] Processing CreateTablet for tablet 0921b09ac9794d89a007640f14283636 (DEFAULT_TABLE table=test-table [id=caf3d629ba2a4bab86e8fbbb6d91b52b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:44:20.302835 2264 tablet_service.cc:1467] Processing CreateTablet for tablet 0921b09ac9794d89a007640f14283636 (DEFAULT_TABLE table=test-table [id=caf3d629ba2a4bab86e8fbbb6d91b52b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:44:20.303470 2404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0921b09ac9794d89a007640f14283636. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:44:20.303817 2264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0921b09ac9794d89a007640f14283636. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:44:20.303869 2544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0921b09ac9794d89a007640f14283636. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:44:20.310561 2633 tablet_bootstrap.cc:492] T 0921b09ac9794d89a007640f14283636 P 50fc1fd1972849d4b96b3df8cc2ca598: Bootstrap starting.
I20260925 04:44:20.311744 2634 tablet_bootstrap.cc:492] T 0921b09ac9794d89a007640f14283636 P 926c2275bff542c48e377eba68599697: Bootstrap starting.
I20260925 04:44:20.312948 2633 tablet_bootstrap.cc:654] T 0921b09ac9794d89a007640f14283636 P 50fc1fd1972849d4b96b3df8cc2ca598: Neither blocks nor log segments found. Creating new log.
I20260925 04:44:20.313501 2634 tablet_bootstrap.cc:654] T 0921b09ac9794d89a007640f14283636 P 926c2275bff542c48e377eba68599697: Neither blocks nor log segments found. Creating new log.
I20260925 04:44:20.313760 2633 log.cc:824] T 0921b09ac9794d89a007640f14283636 P 50fc1fd1972849d4b96b3df8cc2ca598: Log is configured to *not* fsync() on all Append() calls
I20260925 04:44:20.314276 2634 log.cc:824] T 0921b09ac9794d89a007640f14283636 P 926c2275bff542c48e377eba68599697: Log is configured to *not* fsync() on all Append() calls
I20260925 04:44:20.314520 2635 tablet_bootstrap.cc:492] T 0921b09ac9794d89a007640f14283636 P 39f2aab90c2a4055830500e2949241a2: Bootstrap starting.
I20260925 04:44:20.316031 2633 tablet_bootstrap.cc:492] T 0921b09ac9794d89a007640f14283636 P 50fc1fd1972849d4b96b3df8cc2ca598: No bootstrap required, opened a new log
I20260925 04:44:20.316210 2634 tablet_bootstrap.cc:492] T 0921b09ac9794d89a007640f14283636 P 926c2275bff542c48e377eba68599697: No bootstrap required, opened a new log
I20260925 04:44:20.316224 2633 ts_tablet_manager.cc:1397] T 0921b09ac9794d89a007640f14283636 P 50fc1fd1972849d4b96b3df8cc2ca598: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 04:44:20.316408 2634 ts_tablet_manager.cc:1397] T 0921b09ac9794d89a007640f14283636 P 926c2275bff542c48e377eba68599697: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 04:44:20.316851 2635 tablet_bootstrap.cc:654] T 0921b09ac9794d89a007640f14283636 P 39f2aab90c2a4055830500e2949241a2: Neither blocks nor log segments found. Creating new log.
I20260925 04:44:20.317703 2635 log.cc:824] T 0921b09ac9794d89a007640f14283636 P 39f2aab90c2a4055830500e2949241a2: Log is configured to *not* fsync() on all Append() calls
I20260925 04:44:20.319893 2635 tablet_bootstrap.cc:492] T 0921b09ac9794d89a007640f14283636 P 39f2aab90c2a4055830500e2949241a2: No bootstrap required, opened a new log
I20260925 04:44:20.320099 2635 ts_tablet_manager.cc:1397] T 0921b09ac9794d89a007640f14283636 P 39f2aab90c2a4055830500e2949241a2: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 04:44:20.320281 2634 raft_consensus.cc:348] T 0921b09ac9794d89a007640f14283636 P 926c2275bff542c48e377eba68599697 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f2aab90c2a4055830500e2949241a2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46525 } } peers { permanent_uuid: "50fc1fd1972849d4b96b3df8cc2ca598" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38111 } } peers { permanent_uuid: "926c2275bff542c48e377eba68599697" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46207 } }
I20260925 04:44:20.320538 2634 raft_consensus.cc:374] T 0921b09ac9794d89a007640f14283636 P 926c2275bff542c48e377eba68599697 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:44:20.320363 2633 raft_consensus.cc:348] T 0921b09ac9794d89a007640f14283636 P 50fc1fd1972849d4b96b3df8cc2ca598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f2aab90c2a4055830500e2949241a2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46525 } } peers { permanent_uuid: "50fc1fd1972849d4b96b3df8cc2ca598" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38111 } } peers { permanent_uuid: "926c2275bff542c48e377eba68599697" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46207 } }
I20260925 04:44:20.320605 2634 raft_consensus.cc:729] T 0921b09ac9794d89a007640f14283636 P 926c2275bff542c48e377eba68599697 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 926c2275bff542c48e377eba68599697, State: Initialized, Role: FOLLOWER
I20260925 04:44:20.320612 2633 raft_consensus.cc:374] T 0921b09ac9794d89a007640f14283636 P 50fc1fd1972849d4b96b3df8cc2ca598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:44:20.320670 2633 raft_consensus.cc:729] T 0921b09ac9794d89a007640f14283636 P 50fc1fd1972849d4b96b3df8cc2ca598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50fc1fd1972849d4b96b3df8cc2ca598, State: Initialized, Role: FOLLOWER
I20260925 04:44:20.321072 2634 consensus_queue.cc:260] T 0921b09ac9794d89a007640f14283636 P 926c2275bff542c48e377eba68599697 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f2aab90c2a4055830500e2949241a2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46525 } } peers { permanent_uuid: "50fc1fd1972849d4b96b3df8cc2ca598" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38111 } } peers { permanent_uuid: "926c2275bff542c48e377eba68599697" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46207 } }
I20260925 04:44:20.321089 2633 consensus_queue.cc:260] T 0921b09ac9794d89a007640f14283636 P 50fc1fd1972849d4b96b3df8cc2ca598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f2aab90c2a4055830500e2949241a2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46525 } } peers { permanent_uuid: "50fc1fd1972849d4b96b3df8cc2ca598" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38111 } } peers { permanent_uuid: "926c2275bff542c48e377eba68599697" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46207 } }
I20260925 04:44:20.321949 2634 ts_tablet_manager.cc:1428] T 0921b09ac9794d89a007640f14283636 P 926c2275bff542c48e377eba68599697: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260925 04:44:20.322042 2633 ts_tablet_manager.cc:1428] T 0921b09ac9794d89a007640f14283636 P 50fc1fd1972849d4b96b3df8cc2ca598: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 04:44:20.322363 2330 heartbeater.cc:503] Master 127.0.98.126:39315 was elected leader, sending a full tablet report...
I20260925 04:44:20.322383 2610 heartbeater.cc:503] Master 127.0.98.126:39315 was elected leader, sending a full tablet report...
I20260925 04:44:20.324127 2635 raft_consensus.cc:348] T 0921b09ac9794d89a007640f14283636 P 39f2aab90c2a4055830500e2949241a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f2aab90c2a4055830500e2949241a2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46525 } } peers { permanent_uuid: "50fc1fd1972849d4b96b3df8cc2ca598" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38111 } } peers { permanent_uuid: "926c2275bff542c48e377eba68599697" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46207 } }
I20260925 04:44:20.324381 2635 raft_consensus.cc:374] T 0921b09ac9794d89a007640f14283636 P 39f2aab90c2a4055830500e2949241a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:44:20.324466 2635 raft_consensus.cc:729] T 0921b09ac9794d89a007640f14283636 P 39f2aab90c2a4055830500e2949241a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39f2aab90c2a4055830500e2949241a2, State: Initialized, Role: FOLLOWER
I20260925 04:44:20.324900 2635 consensus_queue.cc:260] T 0921b09ac9794d89a007640f14283636 P 39f2aab90c2a4055830500e2949241a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f2aab90c2a4055830500e2949241a2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46525 } } peers { permanent_uuid: "50fc1fd1972849d4b96b3df8cc2ca598" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38111 } } peers { permanent_uuid: "926c2275bff542c48e377eba68599697" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46207 } }
I20260925 04:44:20.325811 2470 heartbeater.cc:503] Master 127.0.98.126:39315 was elected leader, sending a full tablet report...
I20260925 04:44:20.325947 2635 ts_tablet_manager.cc:1428] T 0921b09ac9794d89a007640f14283636 P 39f2aab90c2a4055830500e2949241a2: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 04:44:20.328423 2639 raft_consensus.cc:482] T 0921b09ac9794d89a007640f14283636 P 50fc1fd1972849d4b96b3df8cc2ca598 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:44:20.328595 2639 raft_consensus.cc:504] T 0921b09ac9794d89a007640f14283636 P 50fc1fd1972849d4b96b3df8cc2ca598 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f2aab90c2a4055830500e2949241a2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46525 } } peers { permanent_uuid: "50fc1fd1972849d4b96b3df8cc2ca598" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38111 } } peers { permanent_uuid: "926c2275bff542c48e377eba68599697" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46207 } }
I20260925 04:44:20.329773 2639 leader_election.cc:288] T 0921b09ac9794d89a007640f14283636 P 50fc1fd1972849d4b96b3df8cc2ca598 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39f2aab90c2a4055830500e2949241a2 (127.0.98.66:46525), 926c2275bff542c48e377eba68599697 (127.0.98.65:46207)
I20260925 04:44:20.333701 2503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:20.329959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:46525 (local address 127.0.98.67:37129)
0925 04:44:20.330119 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:20.330136 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:20.330224 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:20.330595 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:20.330598 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:20.330614 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:44:20.330735 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:20.330740 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:20.331458 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:20.331464 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:20.332619 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:20.332628 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:20.333267 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:20.333270 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:20.333369 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:20.333390 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 04:44:20.333491 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 04:44:20.333573 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":1}
I20260925 04:44:20.334163 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:20.330237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:37129 (local address 127.0.98.66:46525)
0925 04:44:20.330379 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:20.330383 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:20.330397 (+ 14us) server_negotiation.cc:407] Connection header received
0925 04:44:20.330440 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:20.330444 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:20.330489 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:20.330572 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:44:20.330839 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:20.331353 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:20.332734 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:20.333148 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:20.333265 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:20.333329 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:20.333999 (+ 670us) server_negotiation.cc:299] Negotiation successful
0925 04:44:20.334077 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 04:44:20.334285 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:20.330422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:58823 (local address 127.0.98.65:46207)
0925 04:44:20.330573 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:20.330580 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:20.330594 (+ 14us) server_negotiation.cc:407] Connection header received
0925 04:44:20.330642 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:20.330644 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:20.330679 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:20.330764 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:44:20.331036 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:20.331523 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:20.333118 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:20.333629 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:20.333752 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:20.333824 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:20.334086 (+ 262us) server_negotiation.cc:299] Negotiation successful
0925 04:44:20.334157 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 04:44:20.334311 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:20.329996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:46207 (local address 127.0.98.67:58823)
0925 04:44:20.330392 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:20.330403 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:20.330504 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:20.330774 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:20.330776 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:20.330791 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:44:20.330923 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:20.330929 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:20.331629 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:20.331635 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:20.332991 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:20.333000 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:20.333758 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:20.333762 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:20.333933 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:20.333955 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 04:44:20.334066 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 04:44:20.334133 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"mutex_wait_us":24,"thread_start_us":122,"threads_started":1}
I20260925 04:44:20.334879 2424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0921b09ac9794d89a007640f14283636" candidate_uuid: "50fc1fd1972849d4b96b3df8cc2ca598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f2aab90c2a4055830500e2949241a2" is_pre_election: true
I20260925 04:44:20.335042 2284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0921b09ac9794d89a007640f14283636" candidate_uuid: "50fc1fd1972849d4b96b3df8cc2ca598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "926c2275bff542c48e377eba68599697" is_pre_election: true
I20260925 04:44:20.335135 2424 raft_consensus.cc:2454] T 0921b09ac9794d89a007640f14283636 P 39f2aab90c2a4055830500e2949241a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50fc1fd1972849d4b96b3df8cc2ca598 in term 0.
I20260925 04:44:20.335341 2284 raft_consensus.cc:2454] T 0921b09ac9794d89a007640f14283636 P 926c2275bff542c48e377eba68599697 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50fc1fd1972849d4b96b3df8cc2ca598 in term 0.
I20260925 04:44:20.335639 2496 leader_election.cc:302] T 0921b09ac9794d89a007640f14283636 P 50fc1fd1972849d4b96b3df8cc2ca598 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39f2aab90c2a4055830500e2949241a2, 50fc1fd1972849d4b96b3df8cc2ca598; no voters:
I20260925 04:44:20.335904 2639 raft_consensus.cc:2781] T 0921b09ac9794d89a007640f14283636 P 50fc1fd1972849d4b96b3df8cc2ca598 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:44:20.335990 2639 raft_consensus.cc:482] T 0921b09ac9794d89a007640f14283636 P 50fc1fd1972849d4b96b3df8cc2ca598 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:44:20.336045 2639 raft_consensus.cc:3037] T 0921b09ac9794d89a007640f14283636 P 50fc1fd1972849d4b96b3df8cc2ca598 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:44:20.337040 2639 raft_consensus.cc:504] T 0921b09ac9794d89a007640f14283636 P 50fc1fd1972849d4b96b3df8cc2ca598 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f2aab90c2a4055830500e2949241a2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46525 } } peers { permanent_uuid: "50fc1fd1972849d4b96b3df8cc2ca598" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38111 } } peers { permanent_uuid: "926c2275bff542c48e377eba68599697" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46207 } }
I20260925 04:44:20.337363 2639 leader_election.cc:288] T 0921b09ac9794d89a007640f14283636 P 50fc1fd1972849d4b96b3df8cc2ca598 [CANDIDATE]: Term 1 election: Requested vote from peers 39f2aab90c2a4055830500e2949241a2 (127.0.98.66:46525), 926c2275bff542c48e377eba68599697 (127.0.98.65:46207)
I20260925 04:44:20.337680 2424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0921b09ac9794d89a007640f14283636" candidate_uuid: "50fc1fd1972849d4b96b3df8cc2ca598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f2aab90c2a4055830500e2949241a2"
I20260925 04:44:20.337682 2284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0921b09ac9794d89a007640f14283636" candidate_uuid: "50fc1fd1972849d4b96b3df8cc2ca598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "926c2275bff542c48e377eba68599697"
I20260925 04:44:20.337810 2284 raft_consensus.cc:3037] T 0921b09ac9794d89a007640f14283636 P 926c2275bff542c48e377eba68599697 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:44:20.337810 2424 raft_consensus.cc:3037] T 0921b09ac9794d89a007640f14283636 P 39f2aab90c2a4055830500e2949241a2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:44:20.339159 2284 raft_consensus.cc:2454] T 0921b09ac9794d89a007640f14283636 P 926c2275bff542c48e377eba68599697 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50fc1fd1972849d4b96b3df8cc2ca598 in term 1.
I20260925 04:44:20.339159 2424 raft_consensus.cc:2454] T 0921b09ac9794d89a007640f14283636 P 39f2aab90c2a4055830500e2949241a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50fc1fd1972849d4b96b3df8cc2ca598 in term 1.
I20260925 04:44:20.339464 2495 leader_election.cc:302] T 0921b09ac9794d89a007640f14283636 P 50fc1fd1972849d4b96b3df8cc2ca598 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50fc1fd1972849d4b96b3df8cc2ca598, 926c2275bff542c48e377eba68599697; no voters:
I20260925 04:44:20.339660 2639 raft_consensus.cc:2781] T 0921b09ac9794d89a007640f14283636 P 50fc1fd1972849d4b96b3df8cc2ca598 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:44:20.339911 2639 raft_consensus.cc:686] T 0921b09ac9794d89a007640f14283636 P 50fc1fd1972849d4b96b3df8cc2ca598 [term 1 LEADER]: Becoming Leader. State: Replica: 50fc1fd1972849d4b96b3df8cc2ca598, State: Running, Role: LEADER
I20260925 04:44:20.340284 2639 consensus_queue.cc:237] T 0921b09ac9794d89a007640f14283636 P 50fc1fd1972849d4b96b3df8cc2ca598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f2aab90c2a4055830500e2949241a2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46525 } } peers { permanent_uuid: "50fc1fd1972849d4b96b3df8cc2ca598" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38111 } } peers { permanent_uuid: "926c2275bff542c48e377eba68599697" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46207 } }
I20260925 04:44:20.343637 2127 catalog_manager.cc:5495] T 0921b09ac9794d89a007640f14283636 P 50fc1fd1972849d4b96b3df8cc2ca598 reported cstate change: term changed from 0 to 1, leader changed from <none> to 50fc1fd1972849d4b96b3df8cc2ca598 (127.0.98.67). New cstate: current_term: 1 leader_uuid: "50fc1fd1972849d4b96b3df8cc2ca598" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f2aab90c2a4055830500e2949241a2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50fc1fd1972849d4b96b3df8cc2ca598" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38111 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "926c2275bff542c48e377eba68599697" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46207 } health_report { overall_health: UNKNOWN } } }
Sep 25 04:44:20 dist-test-slave-kghz krb5kdc[2082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311459, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:44:20.393338 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:20.384248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39315 (local address 127.0.0.1:59754)
0925 04:44:20.384596 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:20.384619 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:20.384729 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:20.385029 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:20.385031 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:20.385432 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:20.385542 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:20.385548 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:20.386362 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:20.386365 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:20.386953 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:20.386961 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:20.387458 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:20.387461 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:20.387531 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:20.388209 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:20.388221 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:20.389781 (+ 1560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:44:20.391523 (+ 1742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:20.391531 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:20.391537 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:20.391802 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:20.392190 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:20.392192 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:20.392193 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:20.392284 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:20.392708 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:44:20.392716 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:44:20.392833 (+ 117us) client_negotiation.cc:769] Sending connection context
0925 04:44:20.392957 (+ 124us) client_negotiation.cc:240] Negotiation successful
0925 04:44:20.393136 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":149,"threads_started":1}
I20260925 04:44:20.393337 2176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:20.384289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59754 (local address 127.0.98.126:39315)
0925 04:44:20.384406 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:20.384410 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:20.384691 (+ 281us) server_negotiation.cc:407] Connection header received
0925 04:44:20.384808 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:20.384810 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:20.384850 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:20.384943 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:20.385661 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:20.386171 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:20.387087 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:20.387299 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:20.387394 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:20.389924 (+ 2530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:20.389927 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:20.389929 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:20.389951 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:20.391426 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:20.391906 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:20.391910 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:20.391912 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:20.392076 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:20.392368 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:20.392371 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:20.392372 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:20.392560 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:20.392694 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:20.393041 (+ 347us) server_negotiation.cc:299] Negotiation successful
0925 04:44:20.393193 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":42}
I20260925 04:44:20.398068 2127 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:59754:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 04:44:20.399035 2127 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:44:20.404263 2264 tablet_service.cc:1467] Processing CreateTablet for tablet 245a7248a839442c90235dfe938d8b4a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=505b6a859303485791ae570c22dacf55]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 04:44:20.404537 2404 tablet_service.cc:1467] Processing CreateTablet for tablet 245a7248a839442c90235dfe938d8b4a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=505b6a859303485791ae570c22dacf55]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 04:44:20.404486 2544 tablet_service.cc:1467] Processing CreateTablet for tablet 245a7248a839442c90235dfe938d8b4a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=505b6a859303485791ae570c22dacf55]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 04:44:20.405036 2264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 245a7248a839442c90235dfe938d8b4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:44:20.405159 2544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 245a7248a839442c90235dfe938d8b4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:44:20.405159 2404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 245a7248a839442c90235dfe938d8b4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:44:20.407624 2634 tablet_bootstrap.cc:492] T 245a7248a839442c90235dfe938d8b4a P 926c2275bff542c48e377eba68599697: Bootstrap starting.
I20260925 04:44:20.407989 2633 tablet_bootstrap.cc:492] T 245a7248a839442c90235dfe938d8b4a P 50fc1fd1972849d4b96b3df8cc2ca598: Bootstrap starting.
I20260925 04:44:20.408197 2635 tablet_bootstrap.cc:492] T 245a7248a839442c90235dfe938d8b4a P 39f2aab90c2a4055830500e2949241a2: Bootstrap starting.
I20260925 04:44:20.408595 2634 tablet_bootstrap.cc:654] T 245a7248a839442c90235dfe938d8b4a P 926c2275bff542c48e377eba68599697: Neither blocks nor log segments found. Creating new log.
I20260925 04:44:20.408986 2633 tablet_bootstrap.cc:654] T 245a7248a839442c90235dfe938d8b4a P 50fc1fd1972849d4b96b3df8cc2ca598: Neither blocks nor log segments found. Creating new log.
I20260925 04:44:20.409117 2635 tablet_bootstrap.cc:654] T 245a7248a839442c90235dfe938d8b4a P 39f2aab90c2a4055830500e2949241a2: Neither blocks nor log segments found. Creating new log.
I20260925 04:44:20.409852 2634 tablet_bootstrap.cc:492] T 245a7248a839442c90235dfe938d8b4a P 926c2275bff542c48e377eba68599697: No bootstrap required, opened a new log
I20260925 04:44:20.409948 2634 ts_tablet_manager.cc:1397] T 245a7248a839442c90235dfe938d8b4a P 926c2275bff542c48e377eba68599697: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 04:44:20.410403 2633 tablet_bootstrap.cc:492] T 245a7248a839442c90235dfe938d8b4a P 50fc1fd1972849d4b96b3df8cc2ca598: No bootstrap required, opened a new log
I20260925 04:44:20.410516 2633 ts_tablet_manager.cc:1397] T 245a7248a839442c90235dfe938d8b4a P 50fc1fd1972849d4b96b3df8cc2ca598: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 04:44:20.410519 2634 raft_consensus.cc:348] T 245a7248a839442c90235dfe938d8b4a P 926c2275bff542c48e377eba68599697 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926c2275bff542c48e377eba68599697" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46207 } } peers { permanent_uuid: "39f2aab90c2a4055830500e2949241a2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46525 } } peers { permanent_uuid: "50fc1fd1972849d4b96b3df8cc2ca598" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38111 } }
I20260925 04:44:20.410651 2634 raft_consensus.cc:374] T 245a7248a839442c90235dfe938d8b4a P 926c2275bff542c48e377eba68599697 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:44:20.410692 2634 raft_consensus.cc:729] T 245a7248a839442c90235dfe938d8b4a P 926c2275bff542c48e377eba68599697 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 926c2275bff542c48e377eba68599697, State: Initialized, Role: FOLLOWER
I20260925 04:44:20.410743 2635 tablet_bootstrap.cc:492] T 245a7248a839442c90235dfe938d8b4a P 39f2aab90c2a4055830500e2949241a2: No bootstrap required, opened a new log
I20260925 04:44:20.410818 2635 ts_tablet_manager.cc:1397] T 245a7248a839442c90235dfe938d8b4a P 39f2aab90c2a4055830500e2949241a2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 04:44:20.410823 2634 consensus_queue.cc:260] T 245a7248a839442c90235dfe938d8b4a P 926c2275bff542c48e377eba68599697 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926c2275bff542c48e377eba68599697" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46207 } } peers { permanent_uuid: "39f2aab90c2a4055830500e2949241a2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46525 } } peers { permanent_uuid: "50fc1fd1972849d4b96b3df8cc2ca598" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38111 } }
I20260925 04:44:20.410975 2633 raft_consensus.cc:348] T 245a7248a839442c90235dfe938d8b4a P 50fc1fd1972849d4b96b3df8cc2ca598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926c2275bff542c48e377eba68599697" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46207 } } peers { permanent_uuid: "39f2aab90c2a4055830500e2949241a2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46525 } } peers { permanent_uuid: "50fc1fd1972849d4b96b3df8cc2ca598" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38111 } }
I20260925 04:44:20.411092 2633 raft_consensus.cc:374] T 245a7248a839442c90235dfe938d8b4a P 50fc1fd1972849d4b96b3df8cc2ca598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:44:20.411126 2633 raft_consensus.cc:729] T 245a7248a839442c90235dfe938d8b4a P 50fc1fd1972849d4b96b3df8cc2ca598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50fc1fd1972849d4b96b3df8cc2ca598, State: Initialized, Role: FOLLOWER
I20260925 04:44:20.411137 2634 ts_tablet_manager.cc:1428] T 245a7248a839442c90235dfe938d8b4a P 926c2275bff542c48e377eba68599697: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260925 04:44:20.411134 2640 tablet_replica.cc:439] T 245a7248a839442c90235dfe938d8b4a P 926c2275bff542c48e377eba68599697: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926c2275bff542c48e377eba68599697" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46207 } } peers { permanent_uuid: "39f2aab90c2a4055830500e2949241a2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46525 } } peers { permanent_uuid: "50fc1fd1972849d4b96b3df8cc2ca598" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38111 } } }
I20260925 04:44:20.411283 2640 tablet_replica.cc:442] T 245a7248a839442c90235dfe938d8b4a P 926c2275bff542c48e377eba68599697: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 04:44:20.411269 2633 consensus_queue.cc:260] T 245a7248a839442c90235dfe938d8b4a P 50fc1fd1972849d4b96b3df8cc2ca598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926c2275bff542c48e377eba68599697" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46207 } } peers { permanent_uuid: "39f2aab90c2a4055830500e2949241a2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46525 } } peers { permanent_uuid: "50fc1fd1972849d4b96b3df8cc2ca598" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38111 } }
I20260925 04:44:20.411266 2635 raft_consensus.cc:348] T 245a7248a839442c90235dfe938d8b4a P 39f2aab90c2a4055830500e2949241a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926c2275bff542c48e377eba68599697" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46207 } } peers { permanent_uuid: "39f2aab90c2a4055830500e2949241a2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46525 } } peers { permanent_uuid: "50fc1fd1972849d4b96b3df8cc2ca598" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38111 } }
I20260925 04:44:20.411368 2635 raft_consensus.cc:374] T 245a7248a839442c90235dfe938d8b4a P 39f2aab90c2a4055830500e2949241a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:44:20.411399 2635 raft_consensus.cc:729] T 245a7248a839442c90235dfe938d8b4a P 39f2aab90c2a4055830500e2949241a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39f2aab90c2a4055830500e2949241a2, State: Initialized, Role: FOLLOWER
I20260925 04:44:20.411520 2639 tablet_replica.cc:439] T 245a7248a839442c90235dfe938d8b4a P 50fc1fd1972849d4b96b3df8cc2ca598: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926c2275bff542c48e377eba68599697" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46207 } } peers { permanent_uuid: "39f2aab90c2a4055830500e2949241a2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46525 } } peers { permanent_uuid: "50fc1fd1972849d4b96b3df8cc2ca598" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38111 } } }
I20260925 04:44:20.411533 2635 consensus_queue.cc:260] T 245a7248a839442c90235dfe938d8b4a P 39f2aab90c2a4055830500e2949241a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926c2275bff542c48e377eba68599697" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46207 } } peers { permanent_uuid: "39f2aab90c2a4055830500e2949241a2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46525 } } peers { permanent_uuid: "50fc1fd1972849d4b96b3df8cc2ca598" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38111 } }
I20260925 04:44:20.411629 2639 tablet_replica.cc:442] T 245a7248a839442c90235dfe938d8b4a P 50fc1fd1972849d4b96b3df8cc2ca598: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 04:44:20.411794 2635 ts_tablet_manager.cc:1428] T 245a7248a839442c90235dfe938d8b4a P 39f2aab90c2a4055830500e2949241a2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 04:44:20.411890 2633 ts_tablet_manager.cc:1428] T 245a7248a839442c90235dfe938d8b4a P 50fc1fd1972849d4b96b3df8cc2ca598: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 04:44:20.411900 2641 tablet_replica.cc:439] T 245a7248a839442c90235dfe938d8b4a P 39f2aab90c2a4055830500e2949241a2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926c2275bff542c48e377eba68599697" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46207 } } peers { permanent_uuid: "39f2aab90c2a4055830500e2949241a2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46525 } } peers { permanent_uuid: "50fc1fd1972849d4b96b3df8cc2ca598" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38111 } } }
I20260925 04:44:20.412025 2641 tablet_replica.cc:442] T 245a7248a839442c90235dfe938d8b4a P 39f2aab90c2a4055830500e2949241a2: This TxnStatusTablet replica's current role is: FOLLOWER
W20260925 04:44:20.485672 2611 tablet.cc:2369] T 0921b09ac9794d89a007640f14283636 P 50fc1fd1972849d4b96b3df8cc2ca598: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 04:44:20.485975 2471 tablet.cc:2369] T 0921b09ac9794d89a007640f14283636 P 39f2aab90c2a4055830500e2949241a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 04:44:20.488198 2331 tablet.cc:2369] T 0921b09ac9794d89a007640f14283636 P 926c2275bff542c48e377eba68599697: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:44:20.719511 2639 consensus_queue.cc:1035] T 0921b09ac9794d89a007640f14283636 P 50fc1fd1972849d4b96b3df8cc2ca598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "926c2275bff542c48e377eba68599697" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:44:20.725548 2639 consensus_queue.cc:1035] T 0921b09ac9794d89a007640f14283636 P 50fc1fd1972849d4b96b3df8cc2ca598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39f2aab90c2a4055830500e2949241a2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:44:20.802045 2641 raft_consensus.cc:482] T 245a7248a839442c90235dfe938d8b4a P 39f2aab90c2a4055830500e2949241a2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:44:20.802187 2641 raft_consensus.cc:504] T 245a7248a839442c90235dfe938d8b4a P 39f2aab90c2a4055830500e2949241a2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926c2275bff542c48e377eba68599697" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46207 } } peers { permanent_uuid: "39f2aab90c2a4055830500e2949241a2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46525 } } peers { permanent_uuid: "50fc1fd1972849d4b96b3df8cc2ca598" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38111 } }
I20260925 04:44:20.803335 2641 leader_election.cc:288] T 245a7248a839442c90235dfe938d8b4a P 39f2aab90c2a4055830500e2949241a2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 926c2275bff542c48e377eba68599697 (127.0.98.65:46207), 50fc1fd1972849d4b96b3df8cc2ca598 (127.0.98.67:38111)
I20260925 04:44:20.807432 2661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:20.803635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:38111 (local address 127.0.98.66:54891)
0925 04:44:20.804037 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:20.804061 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:20.804227 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:20.804499 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:20.804502 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:20.804516 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:44:20.804660 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:20.804666 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:20.805404 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:20.805411 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:20.806299 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:20.806308 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:20.806995 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:20.806998 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:20.807092 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:20.807117 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 04:44:20.807209 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 04:44:20.807284 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"mutex_wait_us":19,"thread_start_us":97,"threads_started":1}
I20260925 04:44:20.807432 2662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:20.803769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:54891 (local address 127.0.98.67:38111)
0925 04:44:20.804070 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:20.804078 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:20.804164 (+ 86us) server_negotiation.cc:407] Connection header received
0925 04:44:20.804338 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:20.804341 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:20.804389 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:20.804491 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:44:20.804767 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:20.805297 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:20.806431 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:20.806861 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:20.806978 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:20.807044 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:20.807222 (+ 178us) server_negotiation.cc:299] Negotiation successful
0925 04:44:20.807288 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":86,"threads_started":1}
I20260925 04:44:20.807741 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:20.803874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:44491 (local address 127.0.98.65:46207)
0925 04:44:20.804111 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:20.804116 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:20.804263 (+ 147us) server_negotiation.cc:407] Connection header received
0925 04:44:20.804440 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:20.804446 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:20.804501 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:20.804595 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:44:20.804871 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:20.805425 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:20.806651 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:20.807098 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:20.807226 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:20.807290 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:20.807513 (+ 223us) server_negotiation.cc:299] Negotiation successful
0925 04:44:20.807618 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 04:44:20.808063 2660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:20.803635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:46207 (local address 127.0.98.66:44491)
0925 04:44:20.804140 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:20.804156 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:20.804317 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:20.804606 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:20.804609 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:20.804624 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:44:20.804758 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:20.804763 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:20.805541 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:20.805548 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:20.806519 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:20.806529 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:20.807228 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:20.807232 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:20.807373 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:20.807391 (+ 18us) client_negotiation.cc:769] Sending connection context
0925 04:44:20.807808 (+ 417us) client_negotiation.cc:240] Negotiation successful
0925 04:44:20.807892 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"mutex_wait_us":45,"thread_start_us":63,"threads_started":1}
I20260925 04:44:20.808017 2564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "245a7248a839442c90235dfe938d8b4a" candidate_uuid: "39f2aab90c2a4055830500e2949241a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50fc1fd1972849d4b96b3df8cc2ca598" is_pre_election: true
I20260925 04:44:20.808267 2564 raft_consensus.cc:2454] T 245a7248a839442c90235dfe938d8b4a P 50fc1fd1972849d4b96b3df8cc2ca598 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39f2aab90c2a4055830500e2949241a2 in term 0.
I20260925 04:44:20.808555 2284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "245a7248a839442c90235dfe938d8b4a" candidate_uuid: "39f2aab90c2a4055830500e2949241a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "926c2275bff542c48e377eba68599697" is_pre_election: true
I20260925 04:44:20.808697 2284 raft_consensus.cc:2454] T 245a7248a839442c90235dfe938d8b4a P 926c2275bff542c48e377eba68599697 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39f2aab90c2a4055830500e2949241a2 in term 0.
I20260925 04:44:20.808753 2357 leader_election.cc:302] T 245a7248a839442c90235dfe938d8b4a P 39f2aab90c2a4055830500e2949241a2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39f2aab90c2a4055830500e2949241a2, 50fc1fd1972849d4b96b3df8cc2ca598; no voters:
I20260925 04:44:20.809028 2641 raft_consensus.cc:2781] T 245a7248a839442c90235dfe938d8b4a P 39f2aab90c2a4055830500e2949241a2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:44:20.809090 2641 raft_consensus.cc:482] T 245a7248a839442c90235dfe938d8b4a P 39f2aab90c2a4055830500e2949241a2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:44:20.809120 2641 raft_consensus.cc:3037] T 245a7248a839442c90235dfe938d8b4a P 39f2aab90c2a4055830500e2949241a2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:44:20.810199 2641 raft_consensus.cc:504] T 245a7248a839442c90235dfe938d8b4a P 39f2aab90c2a4055830500e2949241a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926c2275bff542c48e377eba68599697" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46207 } } peers { permanent_uuid: "39f2aab90c2a4055830500e2949241a2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46525 } } peers { permanent_uuid: "50fc1fd1972849d4b96b3df8cc2ca598" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38111 } }
I20260925 04:44:20.810644 2641 leader_election.cc:288] T 245a7248a839442c90235dfe938d8b4a P 39f2aab90c2a4055830500e2949241a2 [CANDIDATE]: Term 1 election: Requested vote from peers 926c2275bff542c48e377eba68599697 (127.0.98.65:46207), 50fc1fd1972849d4b96b3df8cc2ca598 (127.0.98.67:38111)
I20260925 04:44:20.811015 2284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "245a7248a839442c90235dfe938d8b4a" candidate_uuid: "39f2aab90c2a4055830500e2949241a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "926c2275bff542c48e377eba68599697"
I20260925 04:44:20.811015 2564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "245a7248a839442c90235dfe938d8b4a" candidate_uuid: "39f2aab90c2a4055830500e2949241a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50fc1fd1972849d4b96b3df8cc2ca598"
I20260925 04:44:20.811141 2564 raft_consensus.cc:3037] T 245a7248a839442c90235dfe938d8b4a P 50fc1fd1972849d4b96b3df8cc2ca598 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:44:20.811141 2284 raft_consensus.cc:3037] T 245a7248a839442c90235dfe938d8b4a P 926c2275bff542c48e377eba68599697 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:44:20.812273 2284 raft_consensus.cc:2454] T 245a7248a839442c90235dfe938d8b4a P 926c2275bff542c48e377eba68599697 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39f2aab90c2a4055830500e2949241a2 in term 1.
I20260925 04:44:20.812532 2564 raft_consensus.cc:2454] T 245a7248a839442c90235dfe938d8b4a P 50fc1fd1972849d4b96b3df8cc2ca598 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39f2aab90c2a4055830500e2949241a2 in term 1.
I20260925 04:44:20.812775 2355 leader_election.cc:302] T 245a7248a839442c90235dfe938d8b4a P 39f2aab90c2a4055830500e2949241a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39f2aab90c2a4055830500e2949241a2, 926c2275bff542c48e377eba68599697; no voters:
I20260925 04:44:20.812970 2641 raft_consensus.cc:2781] T 245a7248a839442c90235dfe938d8b4a P 39f2aab90c2a4055830500e2949241a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:44:20.813206 2641 raft_consensus.cc:686] T 245a7248a839442c90235dfe938d8b4a P 39f2aab90c2a4055830500e2949241a2 [term 1 LEADER]: Becoming Leader. State: Replica: 39f2aab90c2a4055830500e2949241a2, State: Running, Role: LEADER
I20260925 04:44:20.813580 2641 consensus_queue.cc:237] T 245a7248a839442c90235dfe938d8b4a P 39f2aab90c2a4055830500e2949241a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926c2275bff542c48e377eba68599697" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46207 } } peers { permanent_uuid: "39f2aab90c2a4055830500e2949241a2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46525 } } peers { permanent_uuid: "50fc1fd1972849d4b96b3df8cc2ca598" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38111 } }
I20260925 04:44:20.815003 2663 tablet_replica.cc:439] T 245a7248a839442c90235dfe938d8b4a P 39f2aab90c2a4055830500e2949241a2: TxnStatusTablet state changed. Reason: New leader 39f2aab90c2a4055830500e2949241a2. Latest consensus state: current_term: 1 leader_uuid: "39f2aab90c2a4055830500e2949241a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926c2275bff542c48e377eba68599697" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46207 } } peers { permanent_uuid: "39f2aab90c2a4055830500e2949241a2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46525 } } peers { permanent_uuid: "50fc1fd1972849d4b96b3df8cc2ca598" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38111 } } }
I20260925 04:44:20.815127 2663 tablet_replica.cc:442] T 245a7248a839442c90235dfe938d8b4a P 39f2aab90c2a4055830500e2949241a2: This TxnStatusTablet replica's current role is: LEADER
I20260925 04:44:20.815387 2665 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 04:44:20.815553 2665 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 04:44:20.816366 2125 catalog_manager.cc:5495] T 245a7248a839442c90235dfe938d8b4a P 39f2aab90c2a4055830500e2949241a2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39f2aab90c2a4055830500e2949241a2 (127.0.98.66). New cstate: current_term: 1 leader_uuid: "39f2aab90c2a4055830500e2949241a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926c2275bff542c48e377eba68599697" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39f2aab90c2a4055830500e2949241a2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50fc1fd1972849d4b96b3df8cc2ca598" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38111 } health_report { overall_health: UNKNOWN } } }
I20260925 04:44:20.870721 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:20.866281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:46525 (local address 127.0.0.1:55858)
0925 04:44:20.866444 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:20.866462 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:20.866595 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:20.867058 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:20.867060 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:20.867067 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 04:44:20.867163 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:20.867168 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:20.868248 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:20.868251 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:20.868856 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:20.868864 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:20.869470 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:20.869473 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:20.869545 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:20.869593 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 04:44:20.870428 (+ 835us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 04:44:20.870443 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 04:44:20.870518 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 04:44:20.870597 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 04:44:20.870771 2668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:20.866437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55858 (local address 127.0.98.66:46525)
0925 04:44:20.866734 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:20.866740 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:20.866754 (+ 14us) server_negotiation.cc:407] Connection header received
0925 04:44:20.866833 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:20.866836 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:20.866900 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:20.867015 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:44:20.867283 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:20.868130 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:20.868976 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:20.869335 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:20.869445 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:20.869741 (+ 296us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:44:20.870324 (+ 583us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:44:20.870443 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:20.870550 (+ 107us) server_negotiation.cc:299] Negotiation successful
0925 04:44:20.870610 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":83,"threads_started":1}
I20260925 04:44:20.876050 2284 raft_consensus.cc:1261] T 245a7248a839442c90235dfe938d8b4a P 926c2275bff542c48e377eba68599697 [term 1 FOLLOWER]: Refusing update from remote peer 39f2aab90c2a4055830500e2949241a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:44:20.876220 2564 raft_consensus.cc:1261] T 245a7248a839442c90235dfe938d8b4a P 50fc1fd1972849d4b96b3df8cc2ca598 [term 1 FOLLOWER]: Refusing update from remote peer 39f2aab90c2a4055830500e2949241a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:44:20.876734 2663 consensus_queue.cc:1035] T 245a7248a839442c90235dfe938d8b4a P 39f2aab90c2a4055830500e2949241a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "926c2275bff542c48e377eba68599697" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:44:20.876940 2641 consensus_queue.cc:1035] T 245a7248a839442c90235dfe938d8b4a P 39f2aab90c2a4055830500e2949241a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50fc1fd1972849d4b96b3df8cc2ca598" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:44:20.880980 2639 tablet_replica.cc:439] T 245a7248a839442c90235dfe938d8b4a P 50fc1fd1972849d4b96b3df8cc2ca598: TxnStatusTablet state changed. Reason: New leader 39f2aab90c2a4055830500e2949241a2. Latest consensus state: current_term: 1 leader_uuid: "39f2aab90c2a4055830500e2949241a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926c2275bff542c48e377eba68599697" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46207 } } peers { permanent_uuid: "39f2aab90c2a4055830500e2949241a2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46525 } } peers { permanent_uuid: "50fc1fd1972849d4b96b3df8cc2ca598" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38111 } } }
I20260925 04:44:20.881115 2639 tablet_replica.cc:442] T 245a7248a839442c90235dfe938d8b4a P 50fc1fd1972849d4b96b3df8cc2ca598: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 04:44:20.881687 2640 tablet_replica.cc:439] T 245a7248a839442c90235dfe938d8b4a P 926c2275bff542c48e377eba68599697: TxnStatusTablet state changed. Reason: New leader 39f2aab90c2a4055830500e2949241a2. Latest consensus state: current_term: 1 leader_uuid: "39f2aab90c2a4055830500e2949241a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926c2275bff542c48e377eba68599697" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46207 } } peers { permanent_uuid: "39f2aab90c2a4055830500e2949241a2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46525 } } peers { permanent_uuid: "50fc1fd1972849d4b96b3df8cc2ca598" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38111 } } }
I20260925 04:44:20.881810 2640 tablet_replica.cc:442] T 245a7248a839442c90235dfe938d8b4a P 926c2275bff542c48e377eba68599697: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 04:44:20.882287 2641 tablet_replica.cc:439] T 245a7248a839442c90235dfe938d8b4a P 39f2aab90c2a4055830500e2949241a2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "39f2aab90c2a4055830500e2949241a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926c2275bff542c48e377eba68599697" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46207 } } peers { permanent_uuid: "39f2aab90c2a4055830500e2949241a2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46525 } } peers { permanent_uuid: "50fc1fd1972849d4b96b3df8cc2ca598" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38111 } } }
I20260925 04:44:20.882407 2641 tablet_replica.cc:442] T 245a7248a839442c90235dfe938d8b4a P 39f2aab90c2a4055830500e2949241a2: This TxnStatusTablet replica's current role is: LEADER
I20260925 04:44:20.883876 2659 mvcc.cc:205] Tried to move back new op lower bound from 7333115743742058496 to 7333115743495434240. Current Snapshot: MvccSnapshot[applied={T|T < 7333115743742058496 or (T in {7333115743742058496})}]
I20260925 04:44:20.884795 2640 tablet_replica.cc:439] T 245a7248a839442c90235dfe938d8b4a P 926c2275bff542c48e377eba68599697: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "39f2aab90c2a4055830500e2949241a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926c2275bff542c48e377eba68599697" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46207 } } peers { permanent_uuid: "39f2aab90c2a4055830500e2949241a2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46525 } } peers { permanent_uuid: "50fc1fd1972849d4b96b3df8cc2ca598" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38111 } } }
I20260925 04:44:20.884918 2640 tablet_replica.cc:442] T 245a7248a839442c90235dfe938d8b4a P 926c2275bff542c48e377eba68599697: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 04:44:20.885838 2657 mvcc.cc:205] Tried to move back new op lower bound from 7333115743742058496 to 7333115743495434240. Current Snapshot: MvccSnapshot[applied={T|T < 7333115743742058496 or (T in {7333115743742058496})}]
I20260925 04:44:20.887537 2663 tablet_replica.cc:439] T 245a7248a839442c90235dfe938d8b4a P 39f2aab90c2a4055830500e2949241a2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "39f2aab90c2a4055830500e2949241a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926c2275bff542c48e377eba68599697" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46207 } } peers { permanent_uuid: "39f2aab90c2a4055830500e2949241a2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46525 } } peers { permanent_uuid: "50fc1fd1972849d4b96b3df8cc2ca598" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38111 } } }
I20260925 04:44:20.887663 2663 tablet_replica.cc:442] T 245a7248a839442c90235dfe938d8b4a P 39f2aab90c2a4055830500e2949241a2: This TxnStatusTablet replica's current role is: LEADER
I20260925 04:44:20.888643 2639 tablet_replica.cc:439] T 245a7248a839442c90235dfe938d8b4a P 50fc1fd1972849d4b96b3df8cc2ca598: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "39f2aab90c2a4055830500e2949241a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926c2275bff542c48e377eba68599697" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46207 } } peers { permanent_uuid: "39f2aab90c2a4055830500e2949241a2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46525 } } peers { permanent_uuid: "50fc1fd1972849d4b96b3df8cc2ca598" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38111 } } }
I20260925 04:44:20.888764 2639 tablet_replica.cc:442] T 245a7248a839442c90235dfe938d8b4a P 50fc1fd1972849d4b96b3df8cc2ca598: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 04:44:20.895431 2662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:20.888383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39124 (local address 127.0.98.67:38111)
0925 04:44:20.888493 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:20.888497 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:20.890092 (+ 1595us) server_negotiation.cc:407] Connection header received
0925 04:44:20.891592 (+ 1500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:20.891596 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:20.891654 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:20.891755 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:44:20.891977 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:20.892799 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:20.893708 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:20.894045 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:20.894148 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:20.894496 (+ 348us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:44:20.895010 (+ 514us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:44:20.895122 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:20.895229 (+ 107us) server_negotiation.cc:299] Negotiation successful
0925 04:44:20.895287 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 04:44:20.906804 2668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:20.903197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:37677 (local address 127.0.98.66:46525)
0925 04:44:20.903343 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:20.903348 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:20.903532 (+ 184us) server_negotiation.cc:407] Connection header received
0925 04:44:20.903673 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:20.903675 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:20.903719 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:20.903786 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:44:20.904022 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:20.904579 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:20.905767 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:20.905977 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:20.906058 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:20.906410 (+ 352us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:44:20.906483 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:44:20.906538 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:20.906649 (+ 111us) server_negotiation.cc:299] Negotiation successful
0925 04:44:20.906695 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 04:44:20.906868 2679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:20.903101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:46525 (local address 127.0.98.67:37677)
0925 04:44:20.903440 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:20.903455 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:20.903567 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:20.903814 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:20.903817 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:20.903824 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 04:44:20.903928 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:20.903934 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:20.904702 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:20.904705 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:20.905648 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:20.905657 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:20.906084 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:20.906088 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:20.906180 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:20.906232 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 04:44:20.906565 (+ 333us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 04:44:20.906571 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 04:44:20.906642 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 04:44:20.906717 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":123,"threads_started":1}
I20260925 04:44:20.934917 2662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:20.931033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:41715 (local address 127.0.98.67:38111)
0925 04:44:20.931150 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:20.931154 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:20.931328 (+ 174us) server_negotiation.cc:407] Connection header received
0925 04:44:20.931634 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:20.931637 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:20.931700 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:20.931766 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:44:20.932019 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:20.932630 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:20.933783 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:20.934013 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:20.934132 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:20.934483 (+ 351us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:44:20.934566 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:44:20.934644 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:20.934729 (+ 85us) server_negotiation.cc:299] Negotiation successful
0925 04:44:20.934775 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 04:44:20.935097 2681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:20.930982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:38111 (local address 127.0.98.66:41715)
0925 04:44:20.931247 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:20.931260 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:20.931377 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:20.931777 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:20.931780 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:20.931787 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 04:44:20.931915 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:20.931921 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:20.932754 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:20.932758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:20.933669 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:20.933678 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:20.934154 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:20.934158 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:20.934299 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:20.934349 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 04:44:20.934648 (+ 299us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 04:44:20.934656 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 04:44:20.934721 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 04:44:20.934800 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":83,"threads_started":1}
I20260925 04:44:20.964684 2125 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:44:20.964884 2125 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:44:20.967633 2125 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4821c63c5d2148a69569f787e8953a81: Sending DeleteTablet for 3 replicas of tablet 0921b09ac9794d89a007640f14283636
I20260925 04:44:20.968386 2404 tablet_service.cc:1514] Processing DeleteTablet for tablet 0921b09ac9794d89a007640f14283636 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:44:20 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:55848
I20260925 04:44:20.968510 2264 tablet_service.cc:1514] Processing DeleteTablet for tablet 0921b09ac9794d89a007640f14283636 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:44:20 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:49514
I20260925 04:44:20.969085 2684 tablet_replica.cc:330] T 0921b09ac9794d89a007640f14283636 P 39f2aab90c2a4055830500e2949241a2: stopping tablet replica
I20260925 04:44:20.969465 2684 raft_consensus.cc:2229] T 0921b09ac9794d89a007640f14283636 P 39f2aab90c2a4055830500e2949241a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:44:20.969722 2684 raft_consensus.cc:2258] T 0921b09ac9794d89a007640f14283636 P 39f2aab90c2a4055830500e2949241a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:44:20.969786 2544 tablet_service.cc:1514] Processing DeleteTablet for tablet 0921b09ac9794d89a007640f14283636 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:44:20 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:39110
I20260925 04:44:20.970355 2125 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59746:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:44:20.970778 2125 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:44:20.971539 2684 ts_tablet_manager.cc:1905] T 0921b09ac9794d89a007640f14283636 P 39f2aab90c2a4055830500e2949241a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:44:20.972388 2686 tablet_replica.cc:330] T 0921b09ac9794d89a007640f14283636 P 50fc1fd1972849d4b96b3df8cc2ca598: stopping tablet replica
I20260925 04:44:20.972694 2686 raft_consensus.cc:2229] T 0921b09ac9794d89a007640f14283636 P 50fc1fd1972849d4b96b3df8cc2ca598 [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:44:20.973043 2686 raft_consensus.cc:2258] T 0921b09ac9794d89a007640f14283636 P 50fc1fd1972849d4b96b3df8cc2ca598 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:44:20.976024 2684 ts_tablet_manager.cc:1918] T 0921b09ac9794d89a007640f14283636 P 39f2aab90c2a4055830500e2949241a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 04:44:20.976118 2544 tablet_service.cc:1467] Processing CreateTablet for tablet b2c5fe40cdf14af9948cd6149cc81f29 (DEFAULT_TABLE table=test-table [id=afce640b2a984df6b598d9eeadb0853f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:44:20.976191 2684 log.cc:1196] T 0921b09ac9794d89a007640f14283636 P 39f2aab90c2a4055830500e2949241a2: Deleting WAL directory at /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/minicluster-data/ts-1/wal/wals/0921b09ac9794d89a007640f14283636
I20260925 04:44:20.976406 2544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2c5fe40cdf14af9948cd6149cc81f29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:44:20.976452 2264 tablet_service.cc:1467] Processing CreateTablet for tablet b2c5fe40cdf14af9948cd6149cc81f29 (DEFAULT_TABLE table=test-table [id=afce640b2a984df6b598d9eeadb0853f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:44:20.976747 2264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2c5fe40cdf14af9948cd6149cc81f29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:44:20.976883 2684 ts_tablet_manager.cc:1939] T 0921b09ac9794d89a007640f14283636 P 39f2aab90c2a4055830500e2949241a2: Deleting consensus metadata
I20260925 04:44:20.978147 2112 catalog_manager.cc:4841] TS 39f2aab90c2a4055830500e2949241a2 (127.0.98.66:46525): tablet 0921b09ac9794d89a007640f14283636 (table test-table [id=caf3d629ba2a4bab86e8fbbb6d91b52b]) successfully deleted
I20260925 04:44:20.978946 2687 tablet_bootstrap.cc:492] T b2c5fe40cdf14af9948cd6149cc81f29 P 50fc1fd1972849d4b96b3df8cc2ca598: Bootstrap starting.
I20260925 04:44:20.979415 2404 tablet_service.cc:1467] Processing CreateTablet for tablet b2c5fe40cdf14af9948cd6149cc81f29 (DEFAULT_TABLE table=test-table [id=afce640b2a984df6b598d9eeadb0853f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:44:20.979684 2404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2c5fe40cdf14af9948cd6149cc81f29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:44:20.980186 2687 tablet_bootstrap.cc:654] T b2c5fe40cdf14af9948cd6149cc81f29 P 50fc1fd1972849d4b96b3df8cc2ca598: Neither blocks nor log segments found. Creating new log.
I20260925 04:44:20.980285 2686 ts_tablet_manager.cc:1905] T 0921b09ac9794d89a007640f14283636 P 50fc1fd1972849d4b96b3df8cc2ca598: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:44:20.981456 2687 tablet_bootstrap.cc:492] T b2c5fe40cdf14af9948cd6149cc81f29 P 50fc1fd1972849d4b96b3df8cc2ca598: No bootstrap required, opened a new log
I20260925 04:44:20.981554 2687 ts_tablet_manager.cc:1397] T b2c5fe40cdf14af9948cd6149cc81f29 P 50fc1fd1972849d4b96b3df8cc2ca598: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 04:44:20.981546 2685 tablet_replica.cc:330] T 0921b09ac9794d89a007640f14283636 P 926c2275bff542c48e377eba68599697: stopping tablet replica
I20260925 04:44:20.981859 2685 raft_consensus.cc:2229] T 0921b09ac9794d89a007640f14283636 P 926c2275bff542c48e377eba68599697 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:44:20.982067 2685 raft_consensus.cc:2258] T 0921b09ac9794d89a007640f14283636 P 926c2275bff542c48e377eba68599697 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:44:20.982017 2687 raft_consensus.cc:348] T b2c5fe40cdf14af9948cd6149cc81f29 P 50fc1fd1972849d4b96b3df8cc2ca598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926c2275bff542c48e377eba68599697" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46207 } } peers { permanent_uuid: "39f2aab90c2a4055830500e2949241a2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46525 } } peers { permanent_uuid: "50fc1fd1972849d4b96b3df8cc2ca598" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38111 } }
I20260925 04:44:20.982137 2687 raft_consensus.cc:374] T b2c5fe40cdf14af9948cd6149cc81f29 P 50fc1fd1972849d4b96b3df8cc2ca598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:44:20.982175 2687 raft_consensus.cc:729] T b2c5fe40cdf14af9948cd6149cc81f29 P 50fc1fd1972849d4b96b3df8cc2ca598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50fc1fd1972849d4b96b3df8cc2ca598, State: Initialized, Role: FOLLOWER
I20260925 04:44:20.982347 2687 consensus_queue.cc:260] T b2c5fe40cdf14af9948cd6149cc81f29 P 50fc1fd1972849d4b96b3df8cc2ca598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926c2275bff542c48e377eba68599697" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46207 } } peers { permanent_uuid: "39f2aab90c2a4055830500e2949241a2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46525 } } peers { permanent_uuid: "50fc1fd1972849d4b96b3df8cc2ca598" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38111 } }
I20260925 04:44:20.982637 2687 ts_tablet_manager.cc:1428] T b2c5fe40cdf14af9948cd6149cc81f29 P 50fc1fd1972849d4b96b3df8cc2ca598: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 04:44:20.984788 2685 ts_tablet_manager.cc:1905] T 0921b09ac9794d89a007640f14283636 P 926c2275bff542c48e377eba68599697: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:44:20.989687 2686 ts_tablet_manager.cc:1918] T 0921b09ac9794d89a007640f14283636 P 50fc1fd1972849d4b96b3df8cc2ca598: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 04:44:20.989784 2686 log.cc:1196] T 0921b09ac9794d89a007640f14283636 P 50fc1fd1972849d4b96b3df8cc2ca598: Deleting WAL directory at /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/minicluster-data/ts-2/wal/wals/0921b09ac9794d89a007640f14283636
I20260925 04:44:20.990079 2686 ts_tablet_manager.cc:1939] T 0921b09ac9794d89a007640f14283636 P 50fc1fd1972849d4b96b3df8cc2ca598: Deleting consensus metadata
I20260925 04:44:20.992452 2685 ts_tablet_manager.cc:1918] T 0921b09ac9794d89a007640f14283636 P 926c2275bff542c48e377eba68599697: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 04:44:20.992555 2685 log.cc:1196] T 0921b09ac9794d89a007640f14283636 P 926c2275bff542c48e377eba68599697: Deleting WAL directory at /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/minicluster-data/ts-0/wal/wals/0921b09ac9794d89a007640f14283636
I20260925 04:44:20.992930 2685 ts_tablet_manager.cc:1939] T 0921b09ac9794d89a007640f14283636 P 926c2275bff542c48e377eba68599697: Deleting consensus metadata
I20260925 04:44:20.994225 2111 catalog_manager.cc:4841] TS 926c2275bff542c48e377eba68599697 (127.0.98.65:46207): tablet 0921b09ac9794d89a007640f14283636 (table test-table [id=caf3d629ba2a4bab86e8fbbb6d91b52b]) successfully deleted
I20260925 04:44:20.994760 2113 catalog_manager.cc:4841] TS 50fc1fd1972849d4b96b3df8cc2ca598 (127.0.98.67:38111): tablet 0921b09ac9794d89a007640f14283636 (table test-table [id=caf3d629ba2a4bab86e8fbbb6d91b52b]) successfully deleted
I20260925 04:44:21.003836 2690 tablet_bootstrap.cc:492] T b2c5fe40cdf14af9948cd6149cc81f29 P 39f2aab90c2a4055830500e2949241a2: Bootstrap starting.
I20260925 04:44:21.005029 2690 tablet_bootstrap.cc:654] T b2c5fe40cdf14af9948cd6149cc81f29 P 39f2aab90c2a4055830500e2949241a2: Neither blocks nor log segments found. Creating new log.
I20260925 04:44:21.005733 2688 tablet_bootstrap.cc:492] T b2c5fe40cdf14af9948cd6149cc81f29 P 926c2275bff542c48e377eba68599697: Bootstrap starting.
I20260925 04:44:21.006719 2688 tablet_bootstrap.cc:654] T b2c5fe40cdf14af9948cd6149cc81f29 P 926c2275bff542c48e377eba68599697: Neither blocks nor log segments found. Creating new log.
I20260925 04:44:21.007992 2688 tablet_bootstrap.cc:492] T b2c5fe40cdf14af9948cd6149cc81f29 P 926c2275bff542c48e377eba68599697: No bootstrap required, opened a new log
I20260925 04:44:21.008085 2688 ts_tablet_manager.cc:1397] T b2c5fe40cdf14af9948cd6149cc81f29 P 926c2275bff542c48e377eba68599697: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 04:44:21.008744 2688 raft_consensus.cc:348] T b2c5fe40cdf14af9948cd6149cc81f29 P 926c2275bff542c48e377eba68599697 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926c2275bff542c48e377eba68599697" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46207 } } peers { permanent_uuid: "39f2aab90c2a4055830500e2949241a2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46525 } } peers { permanent_uuid: "50fc1fd1972849d4b96b3df8cc2ca598" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38111 } }
I20260925 04:44:21.008868 2688 raft_consensus.cc:374] T b2c5fe40cdf14af9948cd6149cc81f29 P 926c2275bff542c48e377eba68599697 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:44:21.008899 2688 raft_consensus.cc:729] T b2c5fe40cdf14af9948cd6149cc81f29 P 926c2275bff542c48e377eba68599697 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 926c2275bff542c48e377eba68599697, State: Initialized, Role: FOLLOWER
I20260925 04:44:21.009045 2688 consensus_queue.cc:260] T b2c5fe40cdf14af9948cd6149cc81f29 P 926c2275bff542c48e377eba68599697 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926c2275bff542c48e377eba68599697" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46207 } } peers { permanent_uuid: "39f2aab90c2a4055830500e2949241a2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46525 } } peers { permanent_uuid: "50fc1fd1972849d4b96b3df8cc2ca598" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38111 } }
I20260925 04:44:21.009748 2688 ts_tablet_manager.cc:1428] T b2c5fe40cdf14af9948cd6149cc81f29 P 926c2275bff542c48e377eba68599697: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260925 04:44:21.010748 2690 tablet_bootstrap.cc:492] T b2c5fe40cdf14af9948cd6149cc81f29 P 39f2aab90c2a4055830500e2949241a2: No bootstrap required, opened a new log
I20260925 04:44:21.010840 2690 ts_tablet_manager.cc:1397] T b2c5fe40cdf14af9948cd6149cc81f29 P 39f2aab90c2a4055830500e2949241a2: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260925 04:44:21.011220 2690 raft_consensus.cc:348] T b2c5fe40cdf14af9948cd6149cc81f29 P 39f2aab90c2a4055830500e2949241a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926c2275bff542c48e377eba68599697" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46207 } } peers { permanent_uuid: "39f2aab90c2a4055830500e2949241a2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46525 } } peers { permanent_uuid: "50fc1fd1972849d4b96b3df8cc2ca598" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38111 } }
I20260925 04:44:21.011338 2690 raft_consensus.cc:374] T b2c5fe40cdf14af9948cd6149cc81f29 P 39f2aab90c2a4055830500e2949241a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:44:21.011370 2690 raft_consensus.cc:729] T b2c5fe40cdf14af9948cd6149cc81f29 P 39f2aab90c2a4055830500e2949241a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39f2aab90c2a4055830500e2949241a2, State: Initialized, Role: FOLLOWER
I20260925 04:44:21.011533 2690 consensus_queue.cc:260] T b2c5fe40cdf14af9948cd6149cc81f29 P 39f2aab90c2a4055830500e2949241a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926c2275bff542c48e377eba68599697" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46207 } } peers { permanent_uuid: "39f2aab90c2a4055830500e2949241a2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46525 } } peers { permanent_uuid: "50fc1fd1972849d4b96b3df8cc2ca598" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38111 } }
I20260925 04:44:21.012956 2690 ts_tablet_manager.cc:1428] T b2c5fe40cdf14af9948cd6149cc81f29 P 39f2aab90c2a4055830500e2949241a2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260925 04:44:21.054003 2643 raft_consensus.cc:482] T b2c5fe40cdf14af9948cd6149cc81f29 P 50fc1fd1972849d4b96b3df8cc2ca598 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:44:21.054169 2643 raft_consensus.cc:504] T b2c5fe40cdf14af9948cd6149cc81f29 P 50fc1fd1972849d4b96b3df8cc2ca598 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926c2275bff542c48e377eba68599697" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46207 } } peers { permanent_uuid: "39f2aab90c2a4055830500e2949241a2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46525 } } peers { permanent_uuid: "50fc1fd1972849d4b96b3df8cc2ca598" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38111 } }
I20260925 04:44:21.054769 2643 leader_election.cc:288] T b2c5fe40cdf14af9948cd6149cc81f29 P 50fc1fd1972849d4b96b3df8cc2ca598 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 926c2275bff542c48e377eba68599697 (127.0.98.65:46207), 39f2aab90c2a4055830500e2949241a2 (127.0.98.66:46525)
I20260925 04:44:21.055207 2424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c5fe40cdf14af9948cd6149cc81f29" candidate_uuid: "50fc1fd1972849d4b96b3df8cc2ca598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f2aab90c2a4055830500e2949241a2" is_pre_election: true
I20260925 04:44:21.055228 2284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c5fe40cdf14af9948cd6149cc81f29" candidate_uuid: "50fc1fd1972849d4b96b3df8cc2ca598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "926c2275bff542c48e377eba68599697" is_pre_election: true
I20260925 04:44:21.055377 2424 raft_consensus.cc:2454] T b2c5fe40cdf14af9948cd6149cc81f29 P 39f2aab90c2a4055830500e2949241a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50fc1fd1972849d4b96b3df8cc2ca598 in term 0.
I20260925 04:44:21.055385 2284 raft_consensus.cc:2454] T b2c5fe40cdf14af9948cd6149cc81f29 P 926c2275bff542c48e377eba68599697 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50fc1fd1972849d4b96b3df8cc2ca598 in term 0.
I20260925 04:44:21.055651 2496 leader_election.cc:302] T b2c5fe40cdf14af9948cd6149cc81f29 P 50fc1fd1972849d4b96b3df8cc2ca598 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39f2aab90c2a4055830500e2949241a2, 50fc1fd1972849d4b96b3df8cc2ca598; no voters:
I20260925 04:44:21.055917 2643 raft_consensus.cc:2781] T b2c5fe40cdf14af9948cd6149cc81f29 P 50fc1fd1972849d4b96b3df8cc2ca598 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:44:21.055989 2643 raft_consensus.cc:482] T b2c5fe40cdf14af9948cd6149cc81f29 P 50fc1fd1972849d4b96b3df8cc2ca598 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:44:21.056043 2643 raft_consensus.cc:3037] T b2c5fe40cdf14af9948cd6149cc81f29 P 50fc1fd1972849d4b96b3df8cc2ca598 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:44:21.057113 2643 raft_consensus.cc:504] T b2c5fe40cdf14af9948cd6149cc81f29 P 50fc1fd1972849d4b96b3df8cc2ca598 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926c2275bff542c48e377eba68599697" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46207 } } peers { permanent_uuid: "39f2aab90c2a4055830500e2949241a2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46525 } } peers { permanent_uuid: "50fc1fd1972849d4b96b3df8cc2ca598" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38111 } }
I20260925 04:44:21.057487 2643 leader_election.cc:288] T b2c5fe40cdf14af9948cd6149cc81f29 P 50fc1fd1972849d4b96b3df8cc2ca598 [CANDIDATE]: Term 1 election: Requested vote from peers 926c2275bff542c48e377eba68599697 (127.0.98.65:46207), 39f2aab90c2a4055830500e2949241a2 (127.0.98.66:46525)
I20260925 04:44:21.057845 2424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c5fe40cdf14af9948cd6149cc81f29" candidate_uuid: "50fc1fd1972849d4b96b3df8cc2ca598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f2aab90c2a4055830500e2949241a2"
I20260925 04:44:21.057989 2424 raft_consensus.cc:3037] T b2c5fe40cdf14af9948cd6149cc81f29 P 39f2aab90c2a4055830500e2949241a2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:44:21.059023 2424 raft_consensus.cc:2454] T b2c5fe40cdf14af9948cd6149cc81f29 P 39f2aab90c2a4055830500e2949241a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50fc1fd1972849d4b96b3df8cc2ca598 in term 1.
I20260925 04:44:21.059325 2496 leader_election.cc:302] T b2c5fe40cdf14af9948cd6149cc81f29 P 50fc1fd1972849d4b96b3df8cc2ca598 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39f2aab90c2a4055830500e2949241a2, 50fc1fd1972849d4b96b3df8cc2ca598; no voters:
I20260925 04:44:21.059303 2284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c5fe40cdf14af9948cd6149cc81f29" candidate_uuid: "50fc1fd1972849d4b96b3df8cc2ca598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "926c2275bff542c48e377eba68599697"
I20260925 04:44:21.059461 2284 raft_consensus.cc:3037] T b2c5fe40cdf14af9948cd6149cc81f29 P 926c2275bff542c48e377eba68599697 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:44:21.059542 2643 raft_consensus.cc:2781] T b2c5fe40cdf14af9948cd6149cc81f29 P 50fc1fd1972849d4b96b3df8cc2ca598 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:44:21.059623 2643 raft_consensus.cc:686] T b2c5fe40cdf14af9948cd6149cc81f29 P 50fc1fd1972849d4b96b3df8cc2ca598 [term 1 LEADER]: Becoming Leader. State: Replica: 50fc1fd1972849d4b96b3df8cc2ca598, State: Running, Role: LEADER
I20260925 04:44:21.059759 2643 consensus_queue.cc:237] T b2c5fe40cdf14af9948cd6149cc81f29 P 50fc1fd1972849d4b96b3df8cc2ca598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926c2275bff542c48e377eba68599697" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46207 } } peers { permanent_uuid: "39f2aab90c2a4055830500e2949241a2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46525 } } peers { permanent_uuid: "50fc1fd1972849d4b96b3df8cc2ca598" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38111 } }
I20260925 04:44:21.060554 2284 raft_consensus.cc:2454] T b2c5fe40cdf14af9948cd6149cc81f29 P 926c2275bff542c48e377eba68599697 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50fc1fd1972849d4b96b3df8cc2ca598 in term 1.
I20260925 04:44:21.061659 2126 catalog_manager.cc:5495] T b2c5fe40cdf14af9948cd6149cc81f29 P 50fc1fd1972849d4b96b3df8cc2ca598 reported cstate change: term changed from 0 to 1, leader changed from <none> to 50fc1fd1972849d4b96b3df8cc2ca598 (127.0.98.67). New cstate: current_term: 1 leader_uuid: "50fc1fd1972849d4b96b3df8cc2ca598" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926c2275bff542c48e377eba68599697" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39f2aab90c2a4055830500e2949241a2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50fc1fd1972849d4b96b3df8cc2ca598" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38111 } health_report { overall_health: HEALTHY } } }
I20260925 04:44:21.085160 2284 raft_consensus.cc:1261] T b2c5fe40cdf14af9948cd6149cc81f29 P 926c2275bff542c48e377eba68599697 [term 1 FOLLOWER]: Refusing update from remote peer 50fc1fd1972849d4b96b3df8cc2ca598: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:44:21.085285 2424 raft_consensus.cc:1261] T b2c5fe40cdf14af9948cd6149cc81f29 P 39f2aab90c2a4055830500e2949241a2 [term 1 FOLLOWER]: Refusing update from remote peer 50fc1fd1972849d4b96b3df8cc2ca598: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:44:21.085613 2643 consensus_queue.cc:1035] T b2c5fe40cdf14af9948cd6149cc81f29 P 50fc1fd1972849d4b96b3df8cc2ca598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "926c2275bff542c48e377eba68599697" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:44:21.085803 2656 consensus_queue.cc:1035] T b2c5fe40cdf14af9948cd6149cc81f29 P 50fc1fd1972849d4b96b3df8cc2ca598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39f2aab90c2a4055830500e2949241a2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 04:44:21.101099 2655 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 04:44:22.655907 2126 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:44:22.656105 2126 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:44:22.659113 2126 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4821c63c5d2148a69569f787e8953a81: Sending DeleteTablet for 3 replicas of tablet b2c5fe40cdf14af9948cd6149cc81f29
I20260925 04:44:22.659970 2544 tablet_service.cc:1514] Processing DeleteTablet for tablet b2c5fe40cdf14af9948cd6149cc81f29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:44:22 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:39110
I20260925 04:44:22.660069 2264 tablet_service.cc:1514] Processing DeleteTablet for tablet b2c5fe40cdf14af9948cd6149cc81f29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:44:22 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:49514
I20260925 04:44:22.660234 2404 tablet_service.cc:1514] Processing DeleteTablet for tablet b2c5fe40cdf14af9948cd6149cc81f29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:44:22 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:55848
I20260925 04:44:22.660452 2719 tablet_replica.cc:330] T b2c5fe40cdf14af9948cd6149cc81f29 P 50fc1fd1972849d4b96b3df8cc2ca598: stopping tablet replica
I20260925 04:44:22.660614 2719 raft_consensus.cc:2229] T b2c5fe40cdf14af9948cd6149cc81f29 P 50fc1fd1972849d4b96b3df8cc2ca598 [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:44:22.660674 2721 tablet_replica.cc:330] T b2c5fe40cdf14af9948cd6149cc81f29 P 39f2aab90c2a4055830500e2949241a2: stopping tablet replica
I20260925 04:44:22.660784 2721 raft_consensus.cc:2229] T b2c5fe40cdf14af9948cd6149cc81f29 P 39f2aab90c2a4055830500e2949241a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:44:22.660845 2719 raft_consensus.cc:2258] T b2c5fe40cdf14af9948cd6149cc81f29 P 50fc1fd1972849d4b96b3df8cc2ca598 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:44:22.660915 2721 raft_consensus.cc:2258] T b2c5fe40cdf14af9948cd6149cc81f29 P 39f2aab90c2a4055830500e2949241a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:44:22.661101 2720 tablet_replica.cc:330] T b2c5fe40cdf14af9948cd6149cc81f29 P 926c2275bff542c48e377eba68599697: stopping tablet replica
I20260925 04:44:22.661240 2720 raft_consensus.cc:2229] T b2c5fe40cdf14af9948cd6149cc81f29 P 926c2275bff542c48e377eba68599697 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:44:22.661375 2720 raft_consensus.cc:2258] T b2c5fe40cdf14af9948cd6149cc81f29 P 926c2275bff542c48e377eba68599697 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:44:22.661950 2719 ts_tablet_manager.cc:1905] T b2c5fe40cdf14af9948cd6149cc81f29 P 50fc1fd1972849d4b96b3df8cc2ca598: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:44:22.661952 2720 ts_tablet_manager.cc:1905] T b2c5fe40cdf14af9948cd6149cc81f29 P 926c2275bff542c48e377eba68599697: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:44:22.664099 2721 ts_tablet_manager.cc:1905] T b2c5fe40cdf14af9948cd6149cc81f29 P 39f2aab90c2a4055830500e2949241a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:44:22.664588 2719 ts_tablet_manager.cc:1918] T b2c5fe40cdf14af9948cd6149cc81f29 P 50fc1fd1972849d4b96b3df8cc2ca598: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 04:44:22.664588 2720 ts_tablet_manager.cc:1918] T b2c5fe40cdf14af9948cd6149cc81f29 P 926c2275bff542c48e377eba68599697: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 04:44:22.664669 2719 log.cc:1196] T b2c5fe40cdf14af9948cd6149cc81f29 P 50fc1fd1972849d4b96b3df8cc2ca598: Deleting WAL directory at /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/minicluster-data/ts-2/wal/wals/b2c5fe40cdf14af9948cd6149cc81f29
I20260925 04:44:22.664669 2720 log.cc:1196] T b2c5fe40cdf14af9948cd6149cc81f29 P 926c2275bff542c48e377eba68599697: Deleting WAL directory at /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/minicluster-data/ts-0/wal/wals/b2c5fe40cdf14af9948cd6149cc81f29
I20260925 04:44:22.664959 2719 ts_tablet_manager.cc:1939] T b2c5fe40cdf14af9948cd6149cc81f29 P 50fc1fd1972849d4b96b3df8cc2ca598: Deleting consensus metadata
I20260925 04:44:22.664959 2720 ts_tablet_manager.cc:1939] T b2c5fe40cdf14af9948cd6149cc81f29 P 926c2275bff542c48e377eba68599697: Deleting consensus metadata
I20260925 04:44:22.665777 2113 catalog_manager.cc:4841] TS 50fc1fd1972849d4b96b3df8cc2ca598 (127.0.98.67:38111): tablet b2c5fe40cdf14af9948cd6149cc81f29 (table test-table [id=afce640b2a984df6b598d9eeadb0853f]) successfully deleted
I20260925 04:44:22.666054 2111 catalog_manager.cc:4841] TS 926c2275bff542c48e377eba68599697 (127.0.98.65:46207): tablet b2c5fe40cdf14af9948cd6149cc81f29 (table test-table [id=afce640b2a984df6b598d9eeadb0853f]) successfully deleted
I20260925 04:44:22.667090 2721 ts_tablet_manager.cc:1918] T b2c5fe40cdf14af9948cd6149cc81f29 P 39f2aab90c2a4055830500e2949241a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 04:44:22.667156 2721 log.cc:1196] T b2c5fe40cdf14af9948cd6149cc81f29 P 39f2aab90c2a4055830500e2949241a2: Deleting WAL directory at /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/minicluster-data/ts-1/wal/wals/b2c5fe40cdf14af9948cd6149cc81f29
I20260925 04:44:22.667415 2721 ts_tablet_manager.cc:1939] T b2c5fe40cdf14af9948cd6149cc81f29 P 39f2aab90c2a4055830500e2949241a2: Deleting consensus metadata
I20260925 04:44:22.668164 2112 catalog_manager.cc:4841] TS 39f2aab90c2a4055830500e2949241a2 (127.0.98.66:46525): tablet b2c5fe40cdf14af9948cd6149cc81f29 (table test-table [id=afce640b2a984df6b598d9eeadb0853f]) successfully deleted
Sep 25 04:44:22 dist-test-slave-kghz krb5kdc[2082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311462, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 04:44:22 dist-test-slave-kghz krb5kdc[2082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311462, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:44:22.689093 2729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:22.679276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59764 (local address 127.0.98.126:39315)
0925 04:44:22.679618 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:22.679628 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:22.679650 (+ 22us) server_negotiation.cc:407] Connection header received
0925 04:44:22.679707 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:22.679710 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:22.679767 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:22.679863 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:22.680586 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:22.681191 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:22.682166 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:22.682505 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:22.682598 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:22.685358 (+ 2760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:22.685361 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:22.685363 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:22.685392 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:22.687156 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:22.687618 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:22.687622 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:22.687623 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:22.687711 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:22.688068 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:22.688070 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:22.688072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:22.688283 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:22.688402 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:22.688673 (+ 271us) server_negotiation.cc:299] Negotiation successful
0925 04:44:22.688832 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":109,"threads_started":1}
I20260925 04:44:22.692157 2126 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59764:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:44:22.692584 2126 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:44:22.697489 2544 tablet_service.cc:1467] Processing CreateTablet for tablet 4d1b74b0afa346449abc8626e1381c0d (DEFAULT_TABLE table=test-table [id=d82cdb3f665944a6b4fd5030e70b75a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:44:22.697477 2264 tablet_service.cc:1467] Processing CreateTablet for tablet 4d1b74b0afa346449abc8626e1381c0d (DEFAULT_TABLE table=test-table [id=d82cdb3f665944a6b4fd5030e70b75a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:44:22.697470 2404 tablet_service.cc:1467] Processing CreateTablet for tablet 4d1b74b0afa346449abc8626e1381c0d (DEFAULT_TABLE table=test-table [id=d82cdb3f665944a6b4fd5030e70b75a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:44:22.697793 2264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d1b74b0afa346449abc8626e1381c0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:44:22.697793 2544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d1b74b0afa346449abc8626e1381c0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:44:22.697795 2404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d1b74b0afa346449abc8626e1381c0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:44:22.700129 2730 tablet_bootstrap.cc:492] T 4d1b74b0afa346449abc8626e1381c0d P 39f2aab90c2a4055830500e2949241a2: Bootstrap starting.
I20260925 04:44:22.700789 2732 tablet_bootstrap.cc:492] T 4d1b74b0afa346449abc8626e1381c0d P 50fc1fd1972849d4b96b3df8cc2ca598: Bootstrap starting.
I20260925 04:44:22.701216 2730 tablet_bootstrap.cc:654] T 4d1b74b0afa346449abc8626e1381c0d P 39f2aab90c2a4055830500e2949241a2: Neither blocks nor log segments found. Creating new log.
I20260925 04:44:22.701643 2732 tablet_bootstrap.cc:654] T 4d1b74b0afa346449abc8626e1381c0d P 50fc1fd1972849d4b96b3df8cc2ca598: Neither blocks nor log segments found. Creating new log.
I20260925 04:44:22.702332 2731 tablet_bootstrap.cc:492] T 4d1b74b0afa346449abc8626e1381c0d P 926c2275bff542c48e377eba68599697: Bootstrap starting.
I20260925 04:44:22.702478 2730 tablet_bootstrap.cc:492] T 4d1b74b0afa346449abc8626e1381c0d P 39f2aab90c2a4055830500e2949241a2: No bootstrap required, opened a new log
I20260925 04:44:22.702577 2730 ts_tablet_manager.cc:1397] T 4d1b74b0afa346449abc8626e1381c0d P 39f2aab90c2a4055830500e2949241a2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 04:44:22.702852 2732 tablet_bootstrap.cc:492] T 4d1b74b0afa346449abc8626e1381c0d P 50fc1fd1972849d4b96b3df8cc2ca598: No bootstrap required, opened a new log
I20260925 04:44:22.702934 2732 ts_tablet_manager.cc:1397] T 4d1b74b0afa346449abc8626e1381c0d P 50fc1fd1972849d4b96b3df8cc2ca598: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 04:44:22.703040 2730 raft_consensus.cc:348] T 4d1b74b0afa346449abc8626e1381c0d P 39f2aab90c2a4055830500e2949241a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926c2275bff542c48e377eba68599697" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46207 } } peers { permanent_uuid: "39f2aab90c2a4055830500e2949241a2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46525 } } peers { permanent_uuid: "50fc1fd1972849d4b96b3df8cc2ca598" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38111 } }
I20260925 04:44:22.703172 2730 raft_consensus.cc:374] T 4d1b74b0afa346449abc8626e1381c0d P 39f2aab90c2a4055830500e2949241a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:44:22.703208 2730 raft_consensus.cc:729] T 4d1b74b0afa346449abc8626e1381c0d P 39f2aab90c2a4055830500e2949241a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39f2aab90c2a4055830500e2949241a2, State: Initialized, Role: FOLLOWER
I20260925 04:44:22.703248 2731 tablet_bootstrap.cc:654] T 4d1b74b0afa346449abc8626e1381c0d P 926c2275bff542c48e377eba68599697: Neither blocks nor log segments found. Creating new log.
I20260925 04:44:22.703363 2730 consensus_queue.cc:260] T 4d1b74b0afa346449abc8626e1381c0d P 39f2aab90c2a4055830500e2949241a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926c2275bff542c48e377eba68599697" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46207 } } peers { permanent_uuid: "39f2aab90c2a4055830500e2949241a2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46525 } } peers { permanent_uuid: "50fc1fd1972849d4b96b3df8cc2ca598" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38111 } }
I20260925 04:44:22.703337 2732 raft_consensus.cc:348] T 4d1b74b0afa346449abc8626e1381c0d P 50fc1fd1972849d4b96b3df8cc2ca598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926c2275bff542c48e377eba68599697" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46207 } } peers { permanent_uuid: "39f2aab90c2a4055830500e2949241a2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46525 } } peers { permanent_uuid: "50fc1fd1972849d4b96b3df8cc2ca598" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38111 } }
I20260925 04:44:22.703456 2732 raft_consensus.cc:374] T 4d1b74b0afa346449abc8626e1381c0d P 50fc1fd1972849d4b96b3df8cc2ca598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:44:22.703495 2732 raft_consensus.cc:729] T 4d1b74b0afa346449abc8626e1381c0d P 50fc1fd1972849d4b96b3df8cc2ca598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50fc1fd1972849d4b96b3df8cc2ca598, State: Initialized, Role: FOLLOWER
I20260925 04:44:22.703653 2732 consensus_queue.cc:260] T 4d1b74b0afa346449abc8626e1381c0d P 50fc1fd1972849d4b96b3df8cc2ca598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926c2275bff542c48e377eba68599697" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46207 } } peers { permanent_uuid: "39f2aab90c2a4055830500e2949241a2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46525 } } peers { permanent_uuid: "50fc1fd1972849d4b96b3df8cc2ca598" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38111 } }
I20260925 04:44:22.704084 2732 ts_tablet_manager.cc:1428] T 4d1b74b0afa346449abc8626e1381c0d P 50fc1fd1972849d4b96b3df8cc2ca598: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 04:44:22.704417 2730 ts_tablet_manager.cc:1428] T 4d1b74b0afa346449abc8626e1381c0d P 39f2aab90c2a4055830500e2949241a2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260925 04:44:22.705284 2731 tablet_bootstrap.cc:492] T 4d1b74b0afa346449abc8626e1381c0d P 926c2275bff542c48e377eba68599697: No bootstrap required, opened a new log
I20260925 04:44:22.705387 2731 ts_tablet_manager.cc:1397] T 4d1b74b0afa346449abc8626e1381c0d P 926c2275bff542c48e377eba68599697: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 04:44:22.705829 2731 raft_consensus.cc:348] T 4d1b74b0afa346449abc8626e1381c0d P 926c2275bff542c48e377eba68599697 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926c2275bff542c48e377eba68599697" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46207 } } peers { permanent_uuid: "39f2aab90c2a4055830500e2949241a2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46525 } } peers { permanent_uuid: "50fc1fd1972849d4b96b3df8cc2ca598" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38111 } }
I20260925 04:44:22.705943 2731 raft_consensus.cc:374] T 4d1b74b0afa346449abc8626e1381c0d P 926c2275bff542c48e377eba68599697 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:44:22.705984 2731 raft_consensus.cc:729] T 4d1b74b0afa346449abc8626e1381c0d P 926c2275bff542c48e377eba68599697 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 926c2275bff542c48e377eba68599697, State: Initialized, Role: FOLLOWER
I20260925 04:44:22.706154 2731 consensus_queue.cc:260] T 4d1b74b0afa346449abc8626e1381c0d P 926c2275bff542c48e377eba68599697 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926c2275bff542c48e377eba68599697" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46207 } } peers { permanent_uuid: "39f2aab90c2a4055830500e2949241a2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46525 } } peers { permanent_uuid: "50fc1fd1972849d4b96b3df8cc2ca598" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38111 } }
I20260925 04:44:22.706602 2731 ts_tablet_manager.cc:1428] T 4d1b74b0afa346449abc8626e1381c0d P 926c2275bff542c48e377eba68599697: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260925 04:44:22.825757 2736 raft_consensus.cc:482] T 4d1b74b0afa346449abc8626e1381c0d P 926c2275bff542c48e377eba68599697 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:44:22.825943 2736 raft_consensus.cc:504] T 4d1b74b0afa346449abc8626e1381c0d P 926c2275bff542c48e377eba68599697 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926c2275bff542c48e377eba68599697" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46207 } } peers { permanent_uuid: "39f2aab90c2a4055830500e2949241a2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46525 } } peers { permanent_uuid: "50fc1fd1972849d4b96b3df8cc2ca598" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38111 } }
I20260925 04:44:22.827195 2736 leader_election.cc:288] T 4d1b74b0afa346449abc8626e1381c0d P 926c2275bff542c48e377eba68599697 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39f2aab90c2a4055830500e2949241a2 (127.0.98.66:46525), 50fc1fd1972849d4b96b3df8cc2ca598 (127.0.98.67:38111)
I20260925 04:44:22.831497 2738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:22.827511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:60607 (local address 127.0.98.66:46525)
0925 04:44:22.827836 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:22.827848 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:22.827870 (+ 22us) server_negotiation.cc:407] Connection header received
0925 04:44:22.828088 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:22.828091 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:22.828133 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:22.828231 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:44:22.828505 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:22.829060 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:22.830185 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:22.830684 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:22.830902 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:22.831011 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:22.831269 (+ 258us) server_negotiation.cc:299] Negotiation successful
0925 04:44:22.831354 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":112,"threads_started":1}
I20260925 04:44:22.831537 2737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:22.827390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:46525 (local address 127.0.98.65:60607)
0925 04:44:22.827793 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:22.827807 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:22.827928 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:22.828243 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:22.828245 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:22.828262 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:44:22.828387 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:22.828392 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:22.829178 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:22.829183 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:22.830052 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:22.830062 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:22.830915 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:22.830923 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:22.831046 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:22.831077 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 04:44:22.831246 (+ 169us) client_negotiation.cc:240] Negotiation successful
0925 04:44:22.831332 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"mutex_wait_us":21,"thread_start_us":138,"threads_started":1}
I20260925 04:44:22.831626 2739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:22.827588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:38111 (local address 127.0.98.65:54387)
0925 04:44:22.827979 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:22.827993 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:22.828107 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:22.828384 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:22.828386 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:22.828402 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:44:22.828521 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:22.828527 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:22.829548 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:22.829551 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:22.830387 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:22.830396 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:22.831140 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:22.831144 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:22.831275 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:22.831293 (+ 18us) client_negotiation.cc:769] Sending connection context
0925 04:44:22.831384 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 04:44:22.831457 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"mutex_wait_us":21,"thread_start_us":119,"threads_started":1}
I20260925 04:44:22.832074 2740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:22.827784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:54387 (local address 127.0.98.67:38111)
0925 04:44:22.828084 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:22.828093 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:22.828106 (+ 13us) server_negotiation.cc:407] Connection header received
0925 04:44:22.828236 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:22.828239 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:22.828278 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:22.828373 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:44:22.828847 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:22.829399 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:22.830524 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:22.830958 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:22.831134 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:22.831206 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:22.831401 (+ 195us) server_negotiation.cc:299] Negotiation successful
0925 04:44:22.831485 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":112,"threads_started":1}
I20260925 04:44:22.832047 2424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d1b74b0afa346449abc8626e1381c0d" candidate_uuid: "926c2275bff542c48e377eba68599697" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f2aab90c2a4055830500e2949241a2" is_pre_election: true
I20260925 04:44:22.832222 2424 raft_consensus.cc:2454] T 4d1b74b0afa346449abc8626e1381c0d P 39f2aab90c2a4055830500e2949241a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 926c2275bff542c48e377eba68599697 in term 0.
I20260925 04:44:22.832532 2564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d1b74b0afa346449abc8626e1381c0d" candidate_uuid: "926c2275bff542c48e377eba68599697" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50fc1fd1972849d4b96b3df8cc2ca598" is_pre_election: true
I20260925 04:44:22.832697 2564 raft_consensus.cc:2454] T 4d1b74b0afa346449abc8626e1381c0d P 50fc1fd1972849d4b96b3df8cc2ca598 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 926c2275bff542c48e377eba68599697 in term 0.
I20260925 04:44:22.832755 2216 leader_election.cc:302] T 4d1b74b0afa346449abc8626e1381c0d P 926c2275bff542c48e377eba68599697 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39f2aab90c2a4055830500e2949241a2, 926c2275bff542c48e377eba68599697; no voters:
I20260925 04:44:22.833117 2736 raft_consensus.cc:2781] T 4d1b74b0afa346449abc8626e1381c0d P 926c2275bff542c48e377eba68599697 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:44:22.833194 2736 raft_consensus.cc:482] T 4d1b74b0afa346449abc8626e1381c0d P 926c2275bff542c48e377eba68599697 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:44:22.833249 2736 raft_consensus.cc:3037] T 4d1b74b0afa346449abc8626e1381c0d P 926c2275bff542c48e377eba68599697 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:44:22.834417 2736 raft_consensus.cc:504] T 4d1b74b0afa346449abc8626e1381c0d P 926c2275bff542c48e377eba68599697 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926c2275bff542c48e377eba68599697" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46207 } } peers { permanent_uuid: "39f2aab90c2a4055830500e2949241a2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46525 } } peers { permanent_uuid: "50fc1fd1972849d4b96b3df8cc2ca598" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38111 } }
I20260925 04:44:22.834813 2736 leader_election.cc:288] T 4d1b74b0afa346449abc8626e1381c0d P 926c2275bff542c48e377eba68599697 [CANDIDATE]: Term 1 election: Requested vote from peers 39f2aab90c2a4055830500e2949241a2 (127.0.98.66:46525), 50fc1fd1972849d4b96b3df8cc2ca598 (127.0.98.67:38111)
I20260925 04:44:22.835301 2564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d1b74b0afa346449abc8626e1381c0d" candidate_uuid: "926c2275bff542c48e377eba68599697" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50fc1fd1972849d4b96b3df8cc2ca598"
I20260925 04:44:22.835299 2424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d1b74b0afa346449abc8626e1381c0d" candidate_uuid: "926c2275bff542c48e377eba68599697" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f2aab90c2a4055830500e2949241a2"
I20260925 04:44:22.835440 2564 raft_consensus.cc:3037] T 4d1b74b0afa346449abc8626e1381c0d P 50fc1fd1972849d4b96b3df8cc2ca598 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:44:22.835470 2424 raft_consensus.cc:3037] T 4d1b74b0afa346449abc8626e1381c0d P 39f2aab90c2a4055830500e2949241a2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:44:22.836416 2564 raft_consensus.cc:2454] T 4d1b74b0afa346449abc8626e1381c0d P 50fc1fd1972849d4b96b3df8cc2ca598 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 926c2275bff542c48e377eba68599697 in term 1.
I20260925 04:44:22.836488 2424 raft_consensus.cc:2454] T 4d1b74b0afa346449abc8626e1381c0d P 39f2aab90c2a4055830500e2949241a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 926c2275bff542c48e377eba68599697 in term 1.
I20260925 04:44:22.836882 2216 leader_election.cc:302] T 4d1b74b0afa346449abc8626e1381c0d P 926c2275bff542c48e377eba68599697 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39f2aab90c2a4055830500e2949241a2, 926c2275bff542c48e377eba68599697; no voters:
I20260925 04:44:22.837126 2736 raft_consensus.cc:2781] T 4d1b74b0afa346449abc8626e1381c0d P 926c2275bff542c48e377eba68599697 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:44:22.837379 2736 raft_consensus.cc:686] T 4d1b74b0afa346449abc8626e1381c0d P 926c2275bff542c48e377eba68599697 [term 1 LEADER]: Becoming Leader. State: Replica: 926c2275bff542c48e377eba68599697, State: Running, Role: LEADER
I20260925 04:44:22.837675 2736 consensus_queue.cc:237] T 4d1b74b0afa346449abc8626e1381c0d P 926c2275bff542c48e377eba68599697 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926c2275bff542c48e377eba68599697" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46207 } } peers { permanent_uuid: "39f2aab90c2a4055830500e2949241a2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46525 } } peers { permanent_uuid: "50fc1fd1972849d4b96b3df8cc2ca598" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38111 } }
I20260925 04:44:22.840114 2126 catalog_manager.cc:5495] T 4d1b74b0afa346449abc8626e1381c0d P 926c2275bff542c48e377eba68599697 reported cstate change: term changed from 0 to 1, leader changed from <none> to 926c2275bff542c48e377eba68599697 (127.0.98.65). New cstate: current_term: 1 leader_uuid: "926c2275bff542c48e377eba68599697" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926c2275bff542c48e377eba68599697" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39f2aab90c2a4055830500e2949241a2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50fc1fd1972849d4b96b3df8cc2ca598" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38111 } health_report { overall_health: UNKNOWN } } }
I20260925 04:44:22.869217 2744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:22.865201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49518 (local address 127.0.98.65:46207)
0925 04:44:22.865515 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:22.865526 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:22.865543 (+ 17us) server_negotiation.cc:407] Connection header received
0925 04:44:22.865630 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:22.865633 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:22.865685 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:22.865820 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:44:22.866091 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:22.866719 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:22.867599 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:22.867856 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:22.867988 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:22.868282 (+ 294us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:44:22.868769 (+ 487us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:44:22.868906 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:22.868985 (+ 79us) server_negotiation.cc:299] Negotiation successful
0925 04:44:22.869042 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":95,"threads_started":1}
I20260925 04:44:22.881409 2747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:22.877093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:46525 (local address 127.0.98.65:47693)
0925 04:44:22.877489 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:22.877503 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:22.877614 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:22.878119 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:22.878122 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:22.878130 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 04:44:22.878296 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:22.878302 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:22.879192 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:22.879195 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:22.880061 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:22.880072 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:22.880553 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:22.880556 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:22.880638 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:22.880698 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 04:44:22.881099 (+ 401us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 04:44:22.881106 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 04:44:22.881187 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 04:44:22.881263 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":146,"threads_started":1}
I20260925 04:44:22.881420 2738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:22.877248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:47693 (local address 127.0.98.66:46525)
0925 04:44:22.877427 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:22.877431 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:22.877574 (+ 143us) server_negotiation.cc:407] Connection header received
0925 04:44:22.877743 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:22.877745 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:22.877837 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:22.877929 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:44:22.878452 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:22.879039 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:22.880192 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:22.880407 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:22.880490 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:22.880861 (+ 371us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:44:22.880987 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:44:22.881056 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:22.881258 (+ 202us) server_negotiation.cc:299] Negotiation successful
0925 04:44:22.881315 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260925 04:44:22.888429 2564 raft_consensus.cc:1261] T 4d1b74b0afa346449abc8626e1381c0d P 50fc1fd1972849d4b96b3df8cc2ca598 [term 1 FOLLOWER]: Refusing update from remote peer 926c2275bff542c48e377eba68599697: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:44:22.888440 2424 raft_consensus.cc:1261] T 4d1b74b0afa346449abc8626e1381c0d P 39f2aab90c2a4055830500e2949241a2 [term 1 FOLLOWER]: Refusing update from remote peer 926c2275bff542c48e377eba68599697: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:44:22.888914 2736 consensus_queue.cc:1035] T 4d1b74b0afa346449abc8626e1381c0d P 926c2275bff542c48e377eba68599697 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50fc1fd1972849d4b96b3df8cc2ca598" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:44:22.889149 2741 consensus_queue.cc:1035] T 4d1b74b0afa346449abc8626e1381c0d P 926c2275bff542c48e377eba68599697 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39f2aab90c2a4055830500e2949241a2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:44:22.907459 2752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:22.903753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:46207 (local address 127.0.98.66:41283)
0925 04:44:22.904042 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:22.904055 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:22.904145 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:22.904323 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:22.904325 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:22.904330 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 04:44:22.904417 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:22.904421 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:22.905092 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:22.905095 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:22.905900 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:22.905911 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:22.906583 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:22.906587 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:22.906687 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:22.906723 (+ 36us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 04:44:22.907159 (+ 436us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 04:44:22.907168 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 04:44:22.907235 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 04:44:22.907323 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":114,"threads_started":1}
I20260925 04:44:22.907459 2744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:22.903793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:41283 (local address 127.0.98.65:46207)
0925 04:44:22.903916 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:22.903920 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:22.904124 (+ 204us) server_negotiation.cc:407] Connection header received
0925 04:44:22.904212 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:22.904214 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:22.904258 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:22.904320 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:44:22.904484 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:22.905003 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:22.906077 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:22.906455 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:22.906586 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:22.906991 (+ 405us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:44:22.907066 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:44:22.907155 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:22.907245 (+ 90us) server_negotiation.cc:299] Negotiation successful
0925 04:44:22.907300 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 04:44:22.930909 2126 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59764:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:44:22.931142 2126 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59764:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:44:22.933521 2126 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4821c63c5d2148a69569f787e8953a81: Sending DeleteTablet for 3 replicas of tablet 4d1b74b0afa346449abc8626e1381c0d
I20260925 04:44:22.934149 2264 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d1b74b0afa346449abc8626e1381c0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:44:22 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:49514
I20260925 04:44:22.934201 2404 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d1b74b0afa346449abc8626e1381c0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:44:22 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:55848
I20260925 04:44:22.934374 2720 tablet_replica.cc:330] T 4d1b74b0afa346449abc8626e1381c0d P 926c2275bff542c48e377eba68599697: stopping tablet replica
I20260925 04:44:22.934502 2720 raft_consensus.cc:2229] T 4d1b74b0afa346449abc8626e1381c0d P 926c2275bff542c48e377eba68599697 [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:44:22.934432 2721 tablet_replica.cc:330] T 4d1b74b0afa346449abc8626e1381c0d P 39f2aab90c2a4055830500e2949241a2: stopping tablet replica
I20260925 04:44:22.934427 2544 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d1b74b0afa346449abc8626e1381c0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:44:22 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:39110
I20260925 04:44:22.934649 2721 raft_consensus.cc:2229] T 4d1b74b0afa346449abc8626e1381c0d P 39f2aab90c2a4055830500e2949241a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:44:22.934708 2719 tablet_replica.cc:330] T 4d1b74b0afa346449abc8626e1381c0d P 50fc1fd1972849d4b96b3df8cc2ca598: stopping tablet replica
I20260925 04:44:22.934778 2721 raft_consensus.cc:2258] T 4d1b74b0afa346449abc8626e1381c0d P 39f2aab90c2a4055830500e2949241a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:44:22.934813 2720 raft_consensus.cc:2258] T 4d1b74b0afa346449abc8626e1381c0d P 926c2275bff542c48e377eba68599697 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:44:22.934813 2719 raft_consensus.cc:2229] T 4d1b74b0afa346449abc8626e1381c0d P 50fc1fd1972849d4b96b3df8cc2ca598 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:44:22.934983 2719 raft_consensus.cc:2258] T 4d1b74b0afa346449abc8626e1381c0d P 50fc1fd1972849d4b96b3df8cc2ca598 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:44:22.935660 2721 ts_tablet_manager.cc:1905] T 4d1b74b0afa346449abc8626e1381c0d P 39f2aab90c2a4055830500e2949241a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:44:22.935751 2719 ts_tablet_manager.cc:1905] T 4d1b74b0afa346449abc8626e1381c0d P 50fc1fd1972849d4b96b3df8cc2ca598: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:44:22.935940 2720 ts_tablet_manager.cc:1905] T 4d1b74b0afa346449abc8626e1381c0d P 926c2275bff542c48e377eba68599697: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:44:22.938489 2721 ts_tablet_manager.cc:1918] T 4d1b74b0afa346449abc8626e1381c0d P 39f2aab90c2a4055830500e2949241a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 04:44:22.938513 2719 ts_tablet_manager.cc:1918] T 4d1b74b0afa346449abc8626e1381c0d P 50fc1fd1972849d4b96b3df8cc2ca598: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 04:44:22.938575 2721 log.cc:1196] T 4d1b74b0afa346449abc8626e1381c0d P 39f2aab90c2a4055830500e2949241a2: Deleting WAL directory at /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/minicluster-data/ts-1/wal/wals/4d1b74b0afa346449abc8626e1381c0d
I20260925 04:44:22.938587 2719 log.cc:1196] T 4d1b74b0afa346449abc8626e1381c0d P 50fc1fd1972849d4b96b3df8cc2ca598: Deleting WAL directory at /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/minicluster-data/ts-2/wal/wals/4d1b74b0afa346449abc8626e1381c0d
I20260925 04:44:22.938832 2721 ts_tablet_manager.cc:1939] T 4d1b74b0afa346449abc8626e1381c0d P 39f2aab90c2a4055830500e2949241a2: Deleting consensus metadata
I20260925 04:44:22.938848 2719 ts_tablet_manager.cc:1939] T 4d1b74b0afa346449abc8626e1381c0d P 50fc1fd1972849d4b96b3df8cc2ca598: Deleting consensus metadata
I20260925 04:44:22.939196 2720 ts_tablet_manager.cc:1918] T 4d1b74b0afa346449abc8626e1381c0d P 926c2275bff542c48e377eba68599697: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 04:44:22.939280 2720 log.cc:1196] T 4d1b74b0afa346449abc8626e1381c0d P 926c2275bff542c48e377eba68599697: Deleting WAL directory at /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/minicluster-data/ts-0/wal/wals/4d1b74b0afa346449abc8626e1381c0d
I20260925 04:44:22.939338 2126 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59764:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 04:44:22.939536 2720 ts_tablet_manager.cc:1939] T 4d1b74b0afa346449abc8626e1381c0d P 926c2275bff542c48e377eba68599697: Deleting consensus metadata
W20260925 04:44:22.939872 2126 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:44:22.940367 2111 catalog_manager.cc:4841] TS 926c2275bff542c48e377eba68599697 (127.0.98.65:46207): tablet 4d1b74b0afa346449abc8626e1381c0d (table test-table [id=d82cdb3f665944a6b4fd5030e70b75a5]) successfully deleted
I20260925 04:44:22.940796 2112 catalog_manager.cc:4841] TS 39f2aab90c2a4055830500e2949241a2 (127.0.98.66:46525): tablet 4d1b74b0afa346449abc8626e1381c0d (table test-table [id=d82cdb3f665944a6b4fd5030e70b75a5]) successfully deleted
I20260925 04:44:22.941045 2113 catalog_manager.cc:4841] TS 50fc1fd1972849d4b96b3df8cc2ca598 (127.0.98.67:38111): tablet 4d1b74b0afa346449abc8626e1381c0d (table test-table [id=d82cdb3f665944a6b4fd5030e70b75a5]) successfully deleted
I20260925 04:44:22.945284 2404 tablet_service.cc:1467] Processing CreateTablet for tablet 6131c98e848c4bb0ad750c5a2be2cbe6 (DEFAULT_TABLE table=test-table [id=563ccf8f919d488ab196e9e91f3819a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:44:22.945303 2264 tablet_service.cc:1467] Processing CreateTablet for tablet 6131c98e848c4bb0ad750c5a2be2cbe6 (DEFAULT_TABLE table=test-table [id=563ccf8f919d488ab196e9e91f3819a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:44:22.945291 2544 tablet_service.cc:1467] Processing CreateTablet for tablet 6131c98e848c4bb0ad750c5a2be2cbe6 (DEFAULT_TABLE table=test-table [id=563ccf8f919d488ab196e9e91f3819a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:44:22.945576 2544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6131c98e848c4bb0ad750c5a2be2cbe6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:44:22.945576 2404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6131c98e848c4bb0ad750c5a2be2cbe6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:44:22.945580 2264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6131c98e848c4bb0ad750c5a2be2cbe6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:44:22.948025 2730 tablet_bootstrap.cc:492] T 6131c98e848c4bb0ad750c5a2be2cbe6 P 39f2aab90c2a4055830500e2949241a2: Bootstrap starting.
I20260925 04:44:22.948429 2731 tablet_bootstrap.cc:492] T 6131c98e848c4bb0ad750c5a2be2cbe6 P 926c2275bff542c48e377eba68599697: Bootstrap starting.
I20260925 04:44:22.948915 2730 tablet_bootstrap.cc:654] T 6131c98e848c4bb0ad750c5a2be2cbe6 P 39f2aab90c2a4055830500e2949241a2: Neither blocks nor log segments found. Creating new log.
I20260925 04:44:22.949335 2731 tablet_bootstrap.cc:654] T 6131c98e848c4bb0ad750c5a2be2cbe6 P 926c2275bff542c48e377eba68599697: Neither blocks nor log segments found. Creating new log.
I20260925 04:44:22.949440 2732 tablet_bootstrap.cc:492] T 6131c98e848c4bb0ad750c5a2be2cbe6 P 50fc1fd1972849d4b96b3df8cc2ca598: Bootstrap starting.
I20260925 04:44:22.950201 2732 tablet_bootstrap.cc:654] T 6131c98e848c4bb0ad750c5a2be2cbe6 P 50fc1fd1972849d4b96b3df8cc2ca598: Neither blocks nor log segments found. Creating new log.
I20260925 04:44:22.950716 2731 tablet_bootstrap.cc:492] T 6131c98e848c4bb0ad750c5a2be2cbe6 P 926c2275bff542c48e377eba68599697: No bootstrap required, opened a new log
I20260925 04:44:22.950798 2731 ts_tablet_manager.cc:1397] T 6131c98e848c4bb0ad750c5a2be2cbe6 P 926c2275bff542c48e377eba68599697: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260925 04:44:22.951196 2732 tablet_bootstrap.cc:492] T 6131c98e848c4bb0ad750c5a2be2cbe6 P 50fc1fd1972849d4b96b3df8cc2ca598: No bootstrap required, opened a new log
I20260925 04:44:22.951256 2732 ts_tablet_manager.cc:1397] T 6131c98e848c4bb0ad750c5a2be2cbe6 P 50fc1fd1972849d4b96b3df8cc2ca598: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 04:44:22.951227 2731 raft_consensus.cc:348] T 6131c98e848c4bb0ad750c5a2be2cbe6 P 926c2275bff542c48e377eba68599697 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926c2275bff542c48e377eba68599697" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46207 } } peers { permanent_uuid: "39f2aab90c2a4055830500e2949241a2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46525 } } peers { permanent_uuid: "50fc1fd1972849d4b96b3df8cc2ca598" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38111 } }
I20260925 04:44:22.951339 2731 raft_consensus.cc:374] T 6131c98e848c4bb0ad750c5a2be2cbe6 P 926c2275bff542c48e377eba68599697 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:44:22.951370 2731 raft_consensus.cc:729] T 6131c98e848c4bb0ad750c5a2be2cbe6 P 926c2275bff542c48e377eba68599697 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 926c2275bff542c48e377eba68599697, State: Initialized, Role: FOLLOWER
I20260925 04:44:22.951501 2731 consensus_queue.cc:260] T 6131c98e848c4bb0ad750c5a2be2cbe6 P 926c2275bff542c48e377eba68599697 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926c2275bff542c48e377eba68599697" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46207 } } peers { permanent_uuid: "39f2aab90c2a4055830500e2949241a2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46525 } } peers { permanent_uuid: "50fc1fd1972849d4b96b3df8cc2ca598" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38111 } }
I20260925 04:44:22.951603 2732 raft_consensus.cc:348] T 6131c98e848c4bb0ad750c5a2be2cbe6 P 50fc1fd1972849d4b96b3df8cc2ca598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926c2275bff542c48e377eba68599697" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46207 } } peers { permanent_uuid: "39f2aab90c2a4055830500e2949241a2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46525 } } peers { permanent_uuid: "50fc1fd1972849d4b96b3df8cc2ca598" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38111 } }
I20260925 04:44:22.951737 2732 raft_consensus.cc:374] T 6131c98e848c4bb0ad750c5a2be2cbe6 P 50fc1fd1972849d4b96b3df8cc2ca598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:44:22.951768 2732 raft_consensus.cc:729] T 6131c98e848c4bb0ad750c5a2be2cbe6 P 50fc1fd1972849d4b96b3df8cc2ca598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50fc1fd1972849d4b96b3df8cc2ca598, State: Initialized, Role: FOLLOWER
I20260925 04:44:22.951768 2731 ts_tablet_manager.cc:1428] T 6131c98e848c4bb0ad750c5a2be2cbe6 P 926c2275bff542c48e377eba68599697: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 04:44:22.951885 2732 consensus_queue.cc:260] T 6131c98e848c4bb0ad750c5a2be2cbe6 P 50fc1fd1972849d4b96b3df8cc2ca598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926c2275bff542c48e377eba68599697" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46207 } } peers { permanent_uuid: "39f2aab90c2a4055830500e2949241a2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46525 } } peers { permanent_uuid: "50fc1fd1972849d4b96b3df8cc2ca598" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38111 } }
I20260925 04:44:22.952164 2732 ts_tablet_manager.cc:1428] T 6131c98e848c4bb0ad750c5a2be2cbe6 P 50fc1fd1972849d4b96b3df8cc2ca598: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 04:44:22.953640 2730 tablet_bootstrap.cc:492] T 6131c98e848c4bb0ad750c5a2be2cbe6 P 39f2aab90c2a4055830500e2949241a2: No bootstrap required, opened a new log
I20260925 04:44:22.953760 2730 ts_tablet_manager.cc:1397] T 6131c98e848c4bb0ad750c5a2be2cbe6 P 39f2aab90c2a4055830500e2949241a2: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260925 04:44:22.954191 2730 raft_consensus.cc:348] T 6131c98e848c4bb0ad750c5a2be2cbe6 P 39f2aab90c2a4055830500e2949241a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926c2275bff542c48e377eba68599697" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46207 } } peers { permanent_uuid: "39f2aab90c2a4055830500e2949241a2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46525 } } peers { permanent_uuid: "50fc1fd1972849d4b96b3df8cc2ca598" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38111 } }
I20260925 04:44:22.954497 2730 raft_consensus.cc:374] T 6131c98e848c4bb0ad750c5a2be2cbe6 P 39f2aab90c2a4055830500e2949241a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:44:22.954550 2730 raft_consensus.cc:729] T 6131c98e848c4bb0ad750c5a2be2cbe6 P 39f2aab90c2a4055830500e2949241a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39f2aab90c2a4055830500e2949241a2, State: Initialized, Role: FOLLOWER
I20260925 04:44:22.954700 2730 consensus_queue.cc:260] T 6131c98e848c4bb0ad750c5a2be2cbe6 P 39f2aab90c2a4055830500e2949241a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926c2275bff542c48e377eba68599697" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46207 } } peers { permanent_uuid: "39f2aab90c2a4055830500e2949241a2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46525 } } peers { permanent_uuid: "50fc1fd1972849d4b96b3df8cc2ca598" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38111 } }
I20260925 04:44:22.954979 2730 ts_tablet_manager.cc:1428] T 6131c98e848c4bb0ad750c5a2be2cbe6 P 39f2aab90c2a4055830500e2949241a2: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260925 04:44:23.089007 2713 raft_consensus.cc:482] T 6131c98e848c4bb0ad750c5a2be2cbe6 P 50fc1fd1972849d4b96b3df8cc2ca598 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:44:23.089180 2713 raft_consensus.cc:504] T 6131c98e848c4bb0ad750c5a2be2cbe6 P 50fc1fd1972849d4b96b3df8cc2ca598 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926c2275bff542c48e377eba68599697" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46207 } } peers { permanent_uuid: "39f2aab90c2a4055830500e2949241a2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46525 } } peers { permanent_uuid: "50fc1fd1972849d4b96b3df8cc2ca598" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38111 } }
I20260925 04:44:23.089716 2713 leader_election.cc:288] T 6131c98e848c4bb0ad750c5a2be2cbe6 P 50fc1fd1972849d4b96b3df8cc2ca598 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 926c2275bff542c48e377eba68599697 (127.0.98.65:46207), 39f2aab90c2a4055830500e2949241a2 (127.0.98.66:46525)
I20260925 04:44:23.090301 2283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6131c98e848c4bb0ad750c5a2be2cbe6" candidate_uuid: "50fc1fd1972849d4b96b3df8cc2ca598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "926c2275bff542c48e377eba68599697" is_pre_election: true
I20260925 04:44:23.090487 2283 raft_consensus.cc:2454] T 6131c98e848c4bb0ad750c5a2be2cbe6 P 926c2275bff542c48e377eba68599697 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50fc1fd1972849d4b96b3df8cc2ca598 in term 0.
I20260925 04:44:23.090353 2424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6131c98e848c4bb0ad750c5a2be2cbe6" candidate_uuid: "50fc1fd1972849d4b96b3df8cc2ca598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f2aab90c2a4055830500e2949241a2" is_pre_election: true
I20260925 04:44:23.090705 2424 raft_consensus.cc:2454] T 6131c98e848c4bb0ad750c5a2be2cbe6 P 39f2aab90c2a4055830500e2949241a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50fc1fd1972849d4b96b3df8cc2ca598 in term 0.
I20260925 04:44:23.090909 2495 leader_election.cc:302] T 6131c98e848c4bb0ad750c5a2be2cbe6 P 50fc1fd1972849d4b96b3df8cc2ca598 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50fc1fd1972849d4b96b3df8cc2ca598, 926c2275bff542c48e377eba68599697; no voters:
I20260925 04:44:23.091107 2713 raft_consensus.cc:2781] T 6131c98e848c4bb0ad750c5a2be2cbe6 P 50fc1fd1972849d4b96b3df8cc2ca598 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:44:23.091179 2713 raft_consensus.cc:482] T 6131c98e848c4bb0ad750c5a2be2cbe6 P 50fc1fd1972849d4b96b3df8cc2ca598 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:44:23.091214 2713 raft_consensus.cc:3037] T 6131c98e848c4bb0ad750c5a2be2cbe6 P 50fc1fd1972849d4b96b3df8cc2ca598 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:44:23.092468 2713 raft_consensus.cc:504] T 6131c98e848c4bb0ad750c5a2be2cbe6 P 50fc1fd1972849d4b96b3df8cc2ca598 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926c2275bff542c48e377eba68599697" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46207 } } peers { permanent_uuid: "39f2aab90c2a4055830500e2949241a2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46525 } } peers { permanent_uuid: "50fc1fd1972849d4b96b3df8cc2ca598" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38111 } }
I20260925 04:44:23.092867 2713 leader_election.cc:288] T 6131c98e848c4bb0ad750c5a2be2cbe6 P 50fc1fd1972849d4b96b3df8cc2ca598 [CANDIDATE]: Term 1 election: Requested vote from peers 926c2275bff542c48e377eba68599697 (127.0.98.65:46207), 39f2aab90c2a4055830500e2949241a2 (127.0.98.66:46525)
I20260925 04:44:23.093441 2283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6131c98e848c4bb0ad750c5a2be2cbe6" candidate_uuid: "50fc1fd1972849d4b96b3df8cc2ca598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "926c2275bff542c48e377eba68599697"
I20260925 04:44:23.093495 2424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6131c98e848c4bb0ad750c5a2be2cbe6" candidate_uuid: "50fc1fd1972849d4b96b3df8cc2ca598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f2aab90c2a4055830500e2949241a2"
I20260925 04:44:23.093609 2283 raft_consensus.cc:3037] T 6131c98e848c4bb0ad750c5a2be2cbe6 P 926c2275bff542c48e377eba68599697 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:44:23.093608 2424 raft_consensus.cc:3037] T 6131c98e848c4bb0ad750c5a2be2cbe6 P 39f2aab90c2a4055830500e2949241a2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:44:23.094861 2283 raft_consensus.cc:2454] T 6131c98e848c4bb0ad750c5a2be2cbe6 P 926c2275bff542c48e377eba68599697 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50fc1fd1972849d4b96b3df8cc2ca598 in term 1.
I20260925 04:44:23.094861 2424 raft_consensus.cc:2454] T 6131c98e848c4bb0ad750c5a2be2cbe6 P 39f2aab90c2a4055830500e2949241a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50fc1fd1972849d4b96b3df8cc2ca598 in term 1.
I20260925 04:44:23.095216 2496 leader_election.cc:302] T 6131c98e848c4bb0ad750c5a2be2cbe6 P 50fc1fd1972849d4b96b3df8cc2ca598 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39f2aab90c2a4055830500e2949241a2, 50fc1fd1972849d4b96b3df8cc2ca598; no voters:
I20260925 04:44:23.095445 2713 raft_consensus.cc:2781] T 6131c98e848c4bb0ad750c5a2be2cbe6 P 50fc1fd1972849d4b96b3df8cc2ca598 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:44:23.095553 2713 raft_consensus.cc:686] T 6131c98e848c4bb0ad750c5a2be2cbe6 P 50fc1fd1972849d4b96b3df8cc2ca598 [term 1 LEADER]: Becoming Leader. State: Replica: 50fc1fd1972849d4b96b3df8cc2ca598, State: Running, Role: LEADER
I20260925 04:44:23.095860 2713 consensus_queue.cc:237] T 6131c98e848c4bb0ad750c5a2be2cbe6 P 50fc1fd1972849d4b96b3df8cc2ca598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926c2275bff542c48e377eba68599697" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46207 } } peers { permanent_uuid: "39f2aab90c2a4055830500e2949241a2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46525 } } peers { permanent_uuid: "50fc1fd1972849d4b96b3df8cc2ca598" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38111 } }
I20260925 04:44:23.097684 2125 catalog_manager.cc:5495] T 6131c98e848c4bb0ad750c5a2be2cbe6 P 50fc1fd1972849d4b96b3df8cc2ca598 reported cstate change: term changed from 0 to 1, leader changed from <none> to 50fc1fd1972849d4b96b3df8cc2ca598 (127.0.98.67). New cstate: current_term: 1 leader_uuid: "50fc1fd1972849d4b96b3df8cc2ca598" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926c2275bff542c48e377eba68599697" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39f2aab90c2a4055830500e2949241a2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50fc1fd1972849d4b96b3df8cc2ca598" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38111 } health_report { overall_health: HEALTHY } } }
I20260925 04:44:23.120102 2740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:23.116920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39126 (local address 127.0.98.67:38111)
0925 04:44:23.117037 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:23.117044 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:23.117131 (+ 87us) server_negotiation.cc:407] Connection header received
0925 04:44:23.117279 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:23.117282 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:23.117335 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:23.117415 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:44:23.117660 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:23.118201 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:23.119062 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:23.119301 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:23.119412 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:23.119654 (+ 242us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:44:23.119735 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:44:23.119812 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:23.119900 (+ 88us) server_negotiation.cc:299] Negotiation successful
0925 04:44:23.119955 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 04:44:23.126554 2283 raft_consensus.cc:1261] T 6131c98e848c4bb0ad750c5a2be2cbe6 P 926c2275bff542c48e377eba68599697 [term 1 FOLLOWER]: Refusing update from remote peer 50fc1fd1972849d4b96b3df8cc2ca598: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:44:23.126534 2424 raft_consensus.cc:1261] T 6131c98e848c4bb0ad750c5a2be2cbe6 P 39f2aab90c2a4055830500e2949241a2 [term 1 FOLLOWER]: Refusing update from remote peer 50fc1fd1972849d4b96b3df8cc2ca598: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:44:23.126921 2713 consensus_queue.cc:1035] T 6131c98e848c4bb0ad750c5a2be2cbe6 P 50fc1fd1972849d4b96b3df8cc2ca598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39f2aab90c2a4055830500e2949241a2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:44:23.127153 2656 consensus_queue.cc:1035] T 6131c98e848c4bb0ad750c5a2be2cbe6 P 50fc1fd1972849d4b96b3df8cc2ca598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "926c2275bff542c48e377eba68599697" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 04:44:23.139858 2702 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 04:44:24.693811 2126 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59764:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:44:24.694001 2126 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59764:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:44:24.696811 2126 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4821c63c5d2148a69569f787e8953a81: Sending DeleteTablet for 3 replicas of tablet 6131c98e848c4bb0ad750c5a2be2cbe6
I20260925 04:44:24.697487 2264 tablet_service.cc:1514] Processing DeleteTablet for tablet 6131c98e848c4bb0ad750c5a2be2cbe6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:44:24 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:49514
I20260925 04:44:24.697525 2404 tablet_service.cc:1514] Processing DeleteTablet for tablet 6131c98e848c4bb0ad750c5a2be2cbe6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:44:24 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:55848
I20260925 04:44:24.697814 2544 tablet_service.cc:1514] Processing DeleteTablet for tablet 6131c98e848c4bb0ad750c5a2be2cbe6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:44:24 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:39110
I20260925 04:44:24.697850 2785 tablet_replica.cc:330] T 6131c98e848c4bb0ad750c5a2be2cbe6 P 926c2275bff542c48e377eba68599697: stopping tablet replica
I20260925 04:44:24.697899 2786 tablet_replica.cc:330] T 6131c98e848c4bb0ad750c5a2be2cbe6 P 39f2aab90c2a4055830500e2949241a2: stopping tablet replica
I20260925 04:44:24.698025 2786 raft_consensus.cc:2229] T 6131c98e848c4bb0ad750c5a2be2cbe6 P 39f2aab90c2a4055830500e2949241a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:44:24.698148 2786 raft_consensus.cc:2258] T 6131c98e848c4bb0ad750c5a2be2cbe6 P 39f2aab90c2a4055830500e2949241a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:44:24.698165 2787 tablet_replica.cc:330] T 6131c98e848c4bb0ad750c5a2be2cbe6 P 50fc1fd1972849d4b96b3df8cc2ca598: stopping tablet replica
I20260925 04:44:24.698158 2785 raft_consensus.cc:2229] T 6131c98e848c4bb0ad750c5a2be2cbe6 P 926c2275bff542c48e377eba68599697 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:44:24.698362 2787 raft_consensus.cc:2229] T 6131c98e848c4bb0ad750c5a2be2cbe6 P 50fc1fd1972849d4b96b3df8cc2ca598 [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:44:24.698500 2785 raft_consensus.cc:2258] T 6131c98e848c4bb0ad750c5a2be2cbe6 P 926c2275bff542c48e377eba68599697 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:44:24.698580 2787 raft_consensus.cc:2258] T 6131c98e848c4bb0ad750c5a2be2cbe6 P 50fc1fd1972849d4b96b3df8cc2ca598 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:44:24.698741 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 2199
I20260925 04:44:24.699659 2786 ts_tablet_manager.cc:1905] T 6131c98e848c4bb0ad750c5a2be2cbe6 P 39f2aab90c2a4055830500e2949241a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:44:24.700150 2787 ts_tablet_manager.cc:1905] T 6131c98e848c4bb0ad750c5a2be2cbe6 P 50fc1fd1972849d4b96b3df8cc2ca598: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:44:24.702042 2786 ts_tablet_manager.cc:1918] T 6131c98e848c4bb0ad750c5a2be2cbe6 P 39f2aab90c2a4055830500e2949241a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 04:44:24.702137 2786 log.cc:1196] T 6131c98e848c4bb0ad750c5a2be2cbe6 P 39f2aab90c2a4055830500e2949241a2: Deleting WAL directory at /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/minicluster-data/ts-1/wal/wals/6131c98e848c4bb0ad750c5a2be2cbe6
I20260925 04:44:24.702498 2786 ts_tablet_manager.cc:1939] T 6131c98e848c4bb0ad750c5a2be2cbe6 P 39f2aab90c2a4055830500e2949241a2: Deleting consensus metadata
I20260925 04:44:24.703243 2787 ts_tablet_manager.cc:1918] T 6131c98e848c4bb0ad750c5a2be2cbe6 P 50fc1fd1972849d4b96b3df8cc2ca598: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 04:44:24.703346 2787 log.cc:1196] T 6131c98e848c4bb0ad750c5a2be2cbe6 P 50fc1fd1972849d4b96b3df8cc2ca598: Deleting WAL directory at /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790311449873301-393-0/minicluster-data/ts-2/wal/wals/6131c98e848c4bb0ad750c5a2be2cbe6
I20260925 04:44:24.703512 2112 catalog_manager.cc:4841] TS 39f2aab90c2a4055830500e2949241a2 (127.0.98.66:46525): tablet 6131c98e848c4bb0ad750c5a2be2cbe6 (table test-table [id=563ccf8f919d488ab196e9e91f3819a1]) successfully deleted
I20260925 04:44:24.703701 2787 ts_tablet_manager.cc:1939] T 6131c98e848c4bb0ad750c5a2be2cbe6 P 50fc1fd1972849d4b96b3df8cc2ca598: Deleting consensus metadata
I20260925 04:44:24.706817 2113 catalog_manager.cc:4841] TS 50fc1fd1972849d4b96b3df8cc2ca598 (127.0.98.67:38111): tablet 6131c98e848c4bb0ad750c5a2be2cbe6 (table test-table [id=563ccf8f919d488ab196e9e91f3819a1]) successfully deleted
W20260925 04:44:24.708695 2111 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.65:46207 (error 108)
I20260925 04:44:24.708943 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 2339
I20260925 04:44:24.710983 2789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:24.710500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:46207 (local address 127.0.0.1:49522)
0925 04:44:24.710797 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:24.710880 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.65:46207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":125,"threads_started":1}
W20260925 04:44:24.711126 2111 catalog_manager.cc:4584] TS 926c2275bff542c48e377eba68599697 (127.0.98.65:46207): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6131c98e848c4bb0ad750c5a2be2cbe6: Network error: Client connection negotiation failed: client connection to 127.0.98.65:46207: connect: Connection refused (error 111)
I20260925 04:44:24.716246 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 2479
I20260925 04:44:24.722962 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 2095
2026-09-25T04:44:24Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6703 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:44:24 dist-test-slave-kghz krb5kdc[2792](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:44:24 dist-test-slave-kghz krb5kdc[2792](info): set up 2 sockets
Sep 25 04:44:24 dist-test-slave-kghz krb5kdc[2792](info): commencing operation
krb5kdc: starting...
W20260925 04:44:25.782505 393 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/krb5kdc/test-user.keytab.
Sep 25 04:44:25 dist-test-slave-kghz krb5kdc[2792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311465, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:44:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:44:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260925 04:44:25.914731 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:35365
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:38681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:35365
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:44:26.039801 2805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:44:26.040199 2805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:44:26.040279 2805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:44:26.044646 2805 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:44:26.044756 2805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:44:26.044795 2805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:44:26.044834 2805 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:44:26.044869 2805 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:44:26.051298 2805 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:38681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:35365
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:35365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:44:26.052536 2805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:44:26.053625 2805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:44:26.062592 2813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:26.062587 2810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:26.062598 2811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:26.063100 2805 server_base.cc:1034] running on GCE node
I20260925 04:44:26.063640 2805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:44:26.064813 2805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:44:26.065989 2805 hybrid_clock.cc:648] HybridClock initialized: now 1790311466065968 us; error 33 us; skew 500 ppm
Sep 25 04:44:26 dist-test-slave-kghz krb5kdc[2792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311466, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:44:26.068960 2805 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260925 04:44:26.069844 2805 webserver.cc:460] Webserver started at http://127.0.98.126:43185/ using document root <none> and password file <none>
I20260925 04:44:26.070505 2805 fs_manager.cc:359] Metadata directory not provided
I20260925 04:44:26.070597 2805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:44:26.070837 2805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:44:26.073020 2805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/minicluster-data/master-0/data/instance:
uuid: "6b8a02c0716041a28d5ae6589430a7bb"
format_stamp: "Formatted at 2026-09-25 04:44:26 on dist-test-slave-kghz"
server_key: "dcd3162cb312a49748c626600df9fe67"
server_key_iv: "f70598129207e495da9265fcc8bb65bc"
server_key_version: "encryptionkey@0"
I20260925 04:44:26.073611 2805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/minicluster-data/master-0/wal/instance:
uuid: "6b8a02c0716041a28d5ae6589430a7bb"
format_stamp: "Formatted at 2026-09-25 04:44:26 on dist-test-slave-kghz"
server_key: "dcd3162cb312a49748c626600df9fe67"
server_key_iv: "f70598129207e495da9265fcc8bb65bc"
server_key_version: "encryptionkey@0"
I20260925 04:44:26.077574 2805 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 04:44:26.080274 2820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:26.081813 2805 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 04:44:26.081979 2805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/minicluster-data/master-0/wal
uuid: "6b8a02c0716041a28d5ae6589430a7bb"
format_stamp: "Formatted at 2026-09-25 04:44:26 on dist-test-slave-kghz"
server_key: "dcd3162cb312a49748c626600df9fe67"
server_key_iv: "f70598129207e495da9265fcc8bb65bc"
server_key_version: "encryptionkey@0"
I20260925 04:44:26.082109 2805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:44:26.111840 2805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:44:26.115487 2805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:44:26.115717 2805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:44:26.123410 2805 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:35365
I20260925 04:44:26.123425 2872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:35365 every 8 connection(s)
I20260925 04:44:26.124441 2805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
I20260925 04:44:26.127454 2873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:44:26.132386 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 2805
I20260925 04:44:26.132551 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/minicluster-data/master-0/wal/instance
I20260925 04:44:26.132820 393 external_mini_cluster.cc:1428] Setting key f6f93c0699388ebd62ec0c4a27d3d44d
I20260925 04:44:26.133863 2873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b8a02c0716041a28d5ae6589430a7bb: Bootstrap starting.
I20260925 04:44:26.136934 2873 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b8a02c0716041a28d5ae6589430a7bb: Neither blocks nor log segments found. Creating new log.
I20260925 04:44:26.137837 2873 log.cc:824] T 00000000000000000000000000000000 P 6b8a02c0716041a28d5ae6589430a7bb: Log is configured to *not* fsync() on all Append() calls
I20260925 04:44:26.140358 2873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b8a02c0716041a28d5ae6589430a7bb: No bootstrap required, opened a new log
Sep 25 04:44:26 dist-test-slave-kghz krb5kdc[2792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311465, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:44:26.144680 2873 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b8a02c0716041a28d5ae6589430a7bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b8a02c0716041a28d5ae6589430a7bb" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 35365 } }
I20260925 04:44:26.145103 2873 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b8a02c0716041a28d5ae6589430a7bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:44:26.145212 2873 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b8a02c0716041a28d5ae6589430a7bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b8a02c0716041a28d5ae6589430a7bb, State: Initialized, Role: FOLLOWER
I20260925 04:44:26.145797 2873 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b8a02c0716041a28d5ae6589430a7bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b8a02c0716041a28d5ae6589430a7bb" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 35365 } }
I20260925 04:44:26.145977 2873 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b8a02c0716041a28d5ae6589430a7bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:44:26.146045 2873 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b8a02c0716041a28d5ae6589430a7bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:44:26.146154 2873 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b8a02c0716041a28d5ae6589430a7bb [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:44:26.147539 2873 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b8a02c0716041a28d5ae6589430a7bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b8a02c0716041a28d5ae6589430a7bb" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 35365 } }
I20260925 04:44:26.147976 2873 leader_election.cc:302] T 00000000000000000000000000000000 P 6b8a02c0716041a28d5ae6589430a7bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b8a02c0716041a28d5ae6589430a7bb; no voters:
I20260925 04:44:26.148343 2873 leader_election.cc:288] T 00000000000000000000000000000000 P 6b8a02c0716041a28d5ae6589430a7bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:44:26.148501 2878 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b8a02c0716041a28d5ae6589430a7bb [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:44:26.148737 2878 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b8a02c0716041a28d5ae6589430a7bb [term 1 LEADER]: Becoming Leader. State: Replica: 6b8a02c0716041a28d5ae6589430a7bb, State: Running, Role: LEADER
I20260925 04:44:26.149134 2878 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b8a02c0716041a28d5ae6589430a7bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b8a02c0716041a28d5ae6589430a7bb" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 35365 } }
I20260925 04:44:26.149693 2873 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b8a02c0716041a28d5ae6589430a7bb [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:44:26.150370 2876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:26.134479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58958 (local address 127.0.98.126:35365)
0925 04:44:26.134939 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:26.134959 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:26.135003 (+ 44us) server_negotiation.cc:407] Connection header received
0925 04:44:26.135962 (+ 959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:26.135989 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:26.136454 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:26.136911 (+ 457us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:26.138008 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:26.139361 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:26.140332 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:26.141050 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:26.141225 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:26.144050 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:26.144057 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:26.144077 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:26.144115 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:26.146500 (+ 2385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:26.147053 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:26.147060 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:26.147067 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:26.147234 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:26.147602 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:26.147605 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:26.147607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:26.148185 (+ 578us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:26.148390 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:26.148914 (+ 524us) server_negotiation.cc:299] Negotiation successful
0925 04:44:26.149258 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":112,"threads_started":1}
I20260925 04:44:26.150899 2879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b8a02c0716041a28d5ae6589430a7bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b8a02c0716041a28d5ae6589430a7bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b8a02c0716041a28d5ae6589430a7bb" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 35365 } } }
I20260925 04:44:26.151041 2879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b8a02c0716041a28d5ae6589430a7bb [sys.catalog]: This master's current role is: LEADER
I20260925 04:44:26.151294 2879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b8a02c0716041a28d5ae6589430a7bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b8a02c0716041a28d5ae6589430a7bb. Latest consensus state: current_term: 1 leader_uuid: "6b8a02c0716041a28d5ae6589430a7bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b8a02c0716041a28d5ae6589430a7bb" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 35365 } } }
I20260925 04:44:26.151392 2879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b8a02c0716041a28d5ae6589430a7bb [sys.catalog]: This master's current role is: LEADER
W20260925 04:44:26.155174 2893 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6b8a02c0716041a28d5ae6589430a7bb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 04:44:26.155288 2893 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 04:44:26.155357 2887 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:44:26.156111 2887 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:44:26.163296 2887 catalog_manager.cc:1347] Generated new cluster ID: 4ec5c4941c674de8a32bbeaad8b6c196
I20260925 04:44:26.163395 2887 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:44:26.178925 2887 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:44:26.180577 2887 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:44:26.192301 2887 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b8a02c0716041a28d5ae6589430a7bb: Generated new TSK 0
I20260925 04:44:26.193337 2887 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/krb5kdc/kudu_127.0.98.65.keytab.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/krb5kdc/kudu_127.0.98.65.keytab.
I20260925 04:44:26.229580 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:0
--local_ip_for_outbound_sockets=127.0.98.65
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:35365
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:38681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:44:26.345683 2899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:44:26.345963 2899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:44:26.346005 2899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:44:26.350696 2899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:44:26.350770 2899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:44:26.350876 2899 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260925 04:44:26.357249 2899 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:38681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:35365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2899
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:44:26.358477 2899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:44:26.359588 2899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:44:26.370126 2905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:26.370146 2904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:26.370146 2907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:26.370613 2899 server_base.cc:1034] running on GCE node
I20260925 04:44:26.371110 2899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:44:26.371845 2899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:44:26.373063 2899 hybrid_clock.cc:648] HybridClock initialized: now 1790311466373033 us; error 46 us; skew 500 ppm
Sep 25 04:44:26 dist-test-slave-kghz krb5kdc[2792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311466, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:44:26.376226 2899 init.cc:377] Logged in from keytab as kudu/127.0.98.65@KRBTEST.COM (short username kudu)
I20260925 04:44:26.377238 2899 webserver.cc:460] Webserver started at http://127.0.98.65:38261/ using document root <none> and password file <none>
I20260925 04:44:26.377815 2899 fs_manager.cc:359] Metadata directory not provided
I20260925 04:44:26.377882 2899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:44:26.378058 2899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:44:26.380354 2899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-0/data/instance:
uuid: "0a23d077542049dbaedac57035fdfb89"
format_stamp: "Formatted at 2026-09-25 04:44:26 on dist-test-slave-kghz"
server_key: "a95ea86903f30ee49932d66db09762cd"
server_key_iv: "4dfef8a1eb932dbd4580b7cad4a8efff"
server_key_version: "encryptionkey@0"
I20260925 04:44:26.380931 2899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-0/wal/instance:
uuid: "0a23d077542049dbaedac57035fdfb89"
format_stamp: "Formatted at 2026-09-25 04:44:26 on dist-test-slave-kghz"
server_key: "a95ea86903f30ee49932d66db09762cd"
server_key_iv: "4dfef8a1eb932dbd4580b7cad4a8efff"
server_key_version: "encryptionkey@0"
I20260925 04:44:26.385062 2899 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 04:44:26.387773 2914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:26.389083 2899 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 04:44:26.389191 2899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-0/wal
uuid: "0a23d077542049dbaedac57035fdfb89"
format_stamp: "Formatted at 2026-09-25 04:44:26 on dist-test-slave-kghz"
server_key: "a95ea86903f30ee49932d66db09762cd"
server_key_iv: "4dfef8a1eb932dbd4580b7cad4a8efff"
server_key_version: "encryptionkey@0"
I20260925 04:44:26.389343 2899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:44:26.406972 2899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:44:26.410614 2899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:44:26.410830 2899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:44:26.411513 2899 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:44:26.412700 2899 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:44:26.412789 2899 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:26.412849 2899 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:44:26.412874 2899 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:26.423345 2899 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:45599
I20260925 04:44:26.423363 3027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:45599 every 8 connection(s)
I20260925 04:44:26.424330 2899 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
I20260925 04:44:26.426913 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 2899
I20260925 04:44:26.427042 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-0/wal/instance
I20260925 04:44:26.427348 393 external_mini_cluster.cc:1428] Setting key 8374824329d924ceb318fc479abd48e7
Sep 25 04:44:26 dist-test-slave-kghz krb5kdc[2792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311466, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.66@KRBTEST.COM" created.
I20260925 04:44:26.441757 2876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:26.426208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:54631 (local address 127.0.98.126:35365)
0925 04:44:26.426398 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:26.426402 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:26.427323 (+ 921us) server_negotiation.cc:407] Connection header received
0925 04:44:26.428854 (+ 1531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:26.428858 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:26.428914 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:26.428999 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:26.430809 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:26.431705 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:26.433777 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:26.434040 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:26.434142 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:26.437575 (+ 3433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:26.437579 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:26.437581 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:26.437609 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:26.439267 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:26.439923 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:26.439926 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:26.439927 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:26.440013 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:26.440414 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:26.440418 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:26.440420 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:26.440633 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:26.440749 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:26.441430 (+ 681us) server_negotiation.cc:299] Negotiation successful
0925 04:44:26.441620 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260925 04:44:26.443063 3030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:26.426598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:35365 (local address 127.0.98.65:54631)
0925 04:44:26.427134 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:26.427195 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:26.428641 (+ 1446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:26.429188 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:26.429198 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:26.429769 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:26.430606 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:26.430631 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:26.431884 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:26.431887 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:26.433603 (+ 1716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:26.433620 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:26.434242 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:26.434271 (+ 29us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:26.434551 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:26.435423 (+ 872us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:26.435455 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:26.437352 (+ 1897us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:44:26.439431 (+ 2079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:26.439439 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:26.439460 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:26.439783 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:26.440119 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:26.440122 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:26.440124 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:26.440303 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:26.440752 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:44:26.440762 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:44:26.441098 (+ 336us) client_negotiation.cc:769] Sending connection context
0925 04:44:26.441325 (+ 227us) client_negotiation.cc:240] Negotiation successful
0925 04:44:26.441718 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":109,"threads_started":1}
I20260925 04:44:26.444703 3028 heartbeater.cc:347] Connected to a master server at 127.0.98.126:35365
I20260925 04:44:26.445019 3028 heartbeater.cc:464] Registering TS with master...
I20260925 04:44:26.445654 3028 heartbeater.cc:511] Master 127.0.98.126:35365 requested a full tablet report, sending...
I20260925 04:44:26.447705 2836 ts_manager.cc:194] Registered new tserver with Master: 0a23d077542049dbaedac57035fdfb89 (127.0.98.65:45599)
I20260925 04:44:26.449481 2836 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.65@KRBTEST.COM'} at 127.0.98.65:54631
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/krb5kdc/kudu_127.0.98.66.keytab.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/krb5kdc/kudu_127.0.98.66.keytab.
I20260925 04:44:26.461927 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:0
--local_ip_for_outbound_sockets=127.0.98.66
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:35365
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:38681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:44:26.576746 3033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:44:26.577075 3033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:44:26.577122 3033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:44:26.581692 3033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:44:26.581772 3033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:44:26.581836 3033 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260925 04:44:26.587919 3033 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:38681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:35365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:44:26.589051 3033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:44:26.590098 3033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:44:26.600790 3038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:26.600896 3039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:26.600781 3041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:26.601177 3033 server_base.cc:1034] running on GCE node
I20260925 04:44:26.601621 3033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:44:26.602399 3033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:44:26.603569 3033 hybrid_clock.cc:648] HybridClock initialized: now 1790311466603556 us; error 32 us; skew 500 ppm
Sep 25 04:44:26 dist-test-slave-kghz krb5kdc[2792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311466, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:44:26.606546 3033 init.cc:377] Logged in from keytab as kudu/127.0.98.66@KRBTEST.COM (short username kudu)
I20260925 04:44:26.607509 3033 webserver.cc:460] Webserver started at http://127.0.98.66:39687/ using document root <none> and password file <none>
I20260925 04:44:26.608155 3033 fs_manager.cc:359] Metadata directory not provided
I20260925 04:44:26.608247 3033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:44:26.608487 3033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:44:26.610735 3033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-1/data/instance:
uuid: "3d4c63c5fca742888026deba18a21097"
format_stamp: "Formatted at 2026-09-25 04:44:26 on dist-test-slave-kghz"
server_key: "94994fce5f97963a76014e86461a1135"
server_key_iv: "6a865e9473eda17df7ef1e48f3dcf52c"
server_key_version: "encryptionkey@0"
I20260925 04:44:26.611573 3033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-1/wal/instance:
uuid: "3d4c63c5fca742888026deba18a21097"
format_stamp: "Formatted at 2026-09-25 04:44:26 on dist-test-slave-kghz"
server_key: "94994fce5f97963a76014e86461a1135"
server_key_iv: "6a865e9473eda17df7ef1e48f3dcf52c"
server_key_version: "encryptionkey@0"
I20260925 04:44:26.617321 3033 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 04:44:26.620430 3048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:26.622560 3033 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:44:26.622692 3033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-1/wal
uuid: "3d4c63c5fca742888026deba18a21097"
format_stamp: "Formatted at 2026-09-25 04:44:26 on dist-test-slave-kghz"
server_key: "94994fce5f97963a76014e86461a1135"
server_key_iv: "6a865e9473eda17df7ef1e48f3dcf52c"
server_key_version: "encryptionkey@0"
I20260925 04:44:26.622859 3033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:44:26.640524 3033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:44:26.644428 3033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:44:26.644749 3033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:44:26.645423 3033 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:44:26.646374 3033 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:44:26.646423 3033 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:26.646500 3033 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:44:26.646541 3033 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:26.658648 3033 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:45199
I20260925 04:44:26.658737 3161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:45199 every 8 connection(s)
I20260925 04:44:26.660039 3033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
I20260925 04:44:26.669240 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 3033
I20260925 04:44:26.669358 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-1/wal/instance
I20260925 04:44:26.669682 393 external_mini_cluster.cc:1428] Setting key beb365e475bdbc105c2b64ac6c303b1f
Sep 25 04:44:26 dist-test-slave-kghz krb5kdc[2792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311466, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:44:26.677114 2876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:26.662870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:35411 (local address 127.0.98.126:35365)
0925 04:44:26.663045 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:26.663050 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:26.664110 (+ 1060us) server_negotiation.cc:407] Connection header received
0925 04:44:26.665559 (+ 1449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:26.665573 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:26.665648 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:26.665736 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:26.667188 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:26.667743 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:26.669056 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:26.669443 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:26.669577 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:26.672833 (+ 3256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:26.672840 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:26.672846 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:26.672880 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:26.674596 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:26.675296 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:26.675298 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:26.675299 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:26.675377 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:26.675831 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:26.675834 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:26.675835 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:26.676020 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:26.676102 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:26.676870 (+ 768us) server_negotiation.cc:299] Negotiation successful
0925 04:44:26.676983 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 04:44:26.678548 3164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:26.663335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:35365 (local address 127.0.98.66:35411)
0925 04:44:26.663889 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:26.663938 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:26.665280 (+ 1342us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:26.665952 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:26.665960 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:26.666448 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:26.667012 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:26.667026 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:26.667933 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:26.667936 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:26.668905 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:26.668916 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:26.669682 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:26.669687 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:26.669919 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:26.670847 (+ 928us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:26.670869 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:26.672651 (+ 1782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:44:26.674722 (+ 2071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:26.674732 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:26.674755 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:26.675191 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:26.675476 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:26.675483 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:26.675488 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:26.675715 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:26.676126 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:44:26.676137 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:44:26.676547 (+ 410us) client_negotiation.cc:769] Sending connection context
0925 04:44:26.676809 (+ 262us) client_negotiation.cc:240] Negotiation successful
0925 04:44:26.677279 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":143,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.67@KRBTEST.COM; defaulting to no policy
I20260925 04:44:26.679965 3162 heartbeater.cc:347] Connected to a master server at 127.0.98.126:35365
I20260925 04:44:26.680251 3162 heartbeater.cc:464] Registering TS with master...
I20260925 04:44:26.680819 3162 heartbeater.cc:511] Master 127.0.98.126:35365 requested a full tablet report, sending...
I20260925 04:44:26.681989 2836 ts_manager.cc:194] Registered new tserver with Master: 3d4c63c5fca742888026deba18a21097 (127.0.98.66:45199)
I20260925 04:44:26.682825 2836 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.66@KRBTEST.COM'} at 127.0.98.66:35411
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/krb5kdc/kudu_127.0.98.67.keytab.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/krb5kdc/kudu_127.0.98.67.keytab.
I20260925 04:44:26.702472 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:0
--local_ip_for_outbound_sockets=127.0.98.67
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:35365
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:38681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:44:26.820250 3167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:44:26.820600 3167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:44:26.820654 3167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:44:26.825719 3167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:44:26.825816 3167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:44:26.825882 3167 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260925 04:44:26.832937 3167 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:38681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:35365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3167
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:44:26.834455 3167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:44:26.835718 3167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:44:26.848294 3172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:26.848301 3173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:26.848294 3175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:26.850028 3167 server_base.cc:1034] running on GCE node
I20260925 04:44:26.850764 3167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:44:26.851673 3167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:44:26.852897 3167 hybrid_clock.cc:648] HybridClock initialized: now 1790311466852885 us; error 38 us; skew 500 ppm
Sep 25 04:44:26 dist-test-slave-kghz krb5kdc[2792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311466, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:44:26.856796 3167 init.cc:377] Logged in from keytab as kudu/127.0.98.67@KRBTEST.COM (short username kudu)
I20260925 04:44:26.858072 3167 webserver.cc:460] Webserver started at http://127.0.98.67:34179/ using document root <none> and password file <none>
I20260925 04:44:26.858829 3167 fs_manager.cc:359] Metadata directory not provided
I20260925 04:44:26.858958 3167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:44:26.859222 3167 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:44:26.861632 3167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-2/data/instance:
uuid: "b8224a07881a48b3b1debafffb492f9a"
format_stamp: "Formatted at 2026-09-25 04:44:26 on dist-test-slave-kghz"
server_key: "804a98984a0953543bee75e9445b1b8e"
server_key_iv: "fa47489d8820aa4ba0686388663e4d3f"
server_key_version: "encryptionkey@0"
I20260925 04:44:26.862406 3167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-2/wal/instance:
uuid: "b8224a07881a48b3b1debafffb492f9a"
format_stamp: "Formatted at 2026-09-25 04:44:26 on dist-test-slave-kghz"
server_key: "804a98984a0953543bee75e9445b1b8e"
server_key_iv: "fa47489d8820aa4ba0686388663e4d3f"
server_key_version: "encryptionkey@0"
I20260925 04:44:26.867421 3167 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 04:44:26.871096 3182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:26.872567 3167 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 04:44:26.872733 3167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-2/wal
uuid: "b8224a07881a48b3b1debafffb492f9a"
format_stamp: "Formatted at 2026-09-25 04:44:26 on dist-test-slave-kghz"
server_key: "804a98984a0953543bee75e9445b1b8e"
server_key_iv: "fa47489d8820aa4ba0686388663e4d3f"
server_key_version: "encryptionkey@0"
I20260925 04:44:26.872890 3167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:44:26.891600 3167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:44:26.895493 3167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:44:26.895740 3167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:44:26.896430 3167 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:44:26.897317 3167 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:44:26.897390 3167 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:26.897466 3167 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:44:26.897526 3167 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:26.908812 3167 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:44965
I20260925 04:44:26.908865 3295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:44965 every 8 connection(s)
I20260925 04:44:26.910072 3167 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
I20260925 04:44:26.920025 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 3167
I20260925 04:44:26.920176 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-2/wal/instance
I20260925 04:44:26.920444 393 external_mini_cluster.cc:1428] Setting key aa60b2b26023797e11c45fc36e7131a4
Sep 25 04:44:26 dist-test-slave-kghz krb5kdc[2792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311466, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:44:26.926789 2876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:26.912325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:53427 (local address 127.0.98.126:35365)
0925 04:44:26.912505 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:26.912530 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:26.913613 (+ 1083us) server_negotiation.cc:407] Connection header received
0925 04:44:26.915391 (+ 1778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:26.915394 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:26.915451 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:26.915536 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:26.917196 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:26.917747 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:26.919242 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:26.919481 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:26.919570 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:26.922820 (+ 3250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:26.922824 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:26.922826 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:26.922853 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:26.924292 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:26.924918 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:26.924926 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:26.924927 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:26.925014 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:26.925464 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:26.925467 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:26.925469 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:26.925671 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:26.925778 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:26.926489 (+ 711us) server_negotiation.cc:299] Negotiation successful
0925 04:44:26.926626 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 04:44:26.927788 3298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:26.912744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:35365 (local address 127.0.98.67:53427)
0925 04:44:26.913335 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:26.913388 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:26.915111 (+ 1723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:26.915945 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:26.915959 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:26.916418 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:26.917010 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:26.917027 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:26.917877 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:26.917880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:26.919093 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:26.919105 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:26.919634 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:26.919637 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:26.919870 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:26.920758 (+ 888us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:26.920781 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:26.922634 (+ 1853us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:44:26.924434 (+ 1800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:26.924442 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:26.924461 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:26.924804 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:26.925112 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:26.925187 (+ 75us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:26.925188 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:26.925358 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:26.925778 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:44:26.925787 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:44:26.926114 (+ 327us) client_negotiation.cc:769] Sending connection context
0925 04:44:26.926434 (+ 320us) client_negotiation.cc:240] Negotiation successful
0925 04:44:26.926810 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":165,"threads_started":1}
I20260925 04:44:26.929141 3296 heartbeater.cc:347] Connected to a master server at 127.0.98.126:35365
I20260925 04:44:26.929423 3296 heartbeater.cc:464] Registering TS with master...
I20260925 04:44:26.929971 3296 heartbeater.cc:511] Master 127.0.98.126:35365 requested a full tablet report, sending...
I20260925 04:44:26.931164 2836 ts_manager.cc:194] Registered new tserver with Master: b8224a07881a48b3b1debafffb492f9a (127.0.98.67:44965)
I20260925 04:44:26.931793 2836 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.67@KRBTEST.COM'} at 127.0.98.67:53427
I20260925 04:44:26.935097 393 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 04:44:26.948707 2876 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:44:26.947052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58974 (local address 127.0.98.126:35365)
0925 04:44:26.947213 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:26.947218 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:26.947334 (+ 116us) server_negotiation.cc:407] Connection header received
0925 04:44:26.947506 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:26.947510 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:26.947560 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:26.947648 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:26.948540 (+ 892us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58974: BlockingRecv error: recv got EOF from 127.0.0.1:58974 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 04:44:26.949913 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 2899
I20260925 04:44:26.955679 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 3033
I20260925 04:44:26.961226 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 3167
I20260925 04:44:26.967161 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 2805
2026-09-25T04:44:26Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2244 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790311449873301-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:44:27 dist-test-slave-kghz krb5kdc[3308](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:44:27 dist-test-slave-kghz krb5kdc[3308](info): set up 2 sockets
Sep 25 04:44:27 dist-test-slave-kghz krb5kdc[3308](info): commencing operation
krb5kdc: starting...
W20260925 04:44:28.050796 393 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790311449873301-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790311449873301-393-0/krb5kdc/test-user.keytab.
Sep 25 04:44:28 dist-test-slave-kghz krb5kdc[3308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:44:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:44:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260925 04:44:28.185240 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790311449873301-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790311449873301-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790311449873301-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:46123
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:34051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:46123
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:44:28.296979 3321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:44:28.297241 3321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:44:28.297281 3321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:44:28.301258 3321 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:44:28.301316 3321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:44:28.301337 3321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:44:28.301353 3321 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:44:28.301368 3321 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:44:28.306790 3321 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:34051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790311449873301-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790311449873301-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:46123
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:46123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3321
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790311449873301-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:44:28.307842 3321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:44:28.308830 3321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:44:28.317683 3326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:28.317672 3327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:28.317759 3321 server_base.cc:1034] running on GCE node
W20260925 04:44:28.317679 3329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:28.318557 3321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:44:28.319664 3321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:44:28.320849 3321 hybrid_clock.cc:648] HybridClock initialized: now 1790311468320846 us; error 52 us; skew 500 ppm
Sep 25 04:44:28 dist-test-slave-kghz krb5kdc[3308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311468, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:44:28.330195 3321 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260925 04:44:28.331180 3321 webserver.cc:460] Webserver started at http://127.0.98.126:37753/ using document root <none> and password file <none>
I20260925 04:44:28.331763 3321 fs_manager.cc:359] Metadata directory not provided
I20260925 04:44:28.331823 3321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:44:28.331990 3321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:44:28.334003 3321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790311449873301-393-0/minicluster-data/master-0/data/instance:
uuid: "8f37f800b8494dc3992a62c504b30076"
format_stamp: "Formatted at 2026-09-25 04:44:28 on dist-test-slave-kghz"
server_key: "40d941b9b4a974f94dff22aeb1c9262b"
server_key_iv: "e21d070357a87c2da60def31cd3abfd5"
server_key_version: "encryptionkey@0"
I20260925 04:44:28.334591 3321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790311449873301-393-0/minicluster-data/master-0/wal/instance:
uuid: "8f37f800b8494dc3992a62c504b30076"
format_stamp: "Formatted at 2026-09-25 04:44:28 on dist-test-slave-kghz"
server_key: "40d941b9b4a974f94dff22aeb1c9262b"
server_key_iv: "e21d070357a87c2da60def31cd3abfd5"
server_key_version: "encryptionkey@0"
I20260925 04:44:28.338325 3321 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 04:44:28.340770 3336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:28.342177 3321 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 04:44:28.342361 3321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790311449873301-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790311449873301-393-0/minicluster-data/master-0/wal
uuid: "8f37f800b8494dc3992a62c504b30076"
format_stamp: "Formatted at 2026-09-25 04:44:28 on dist-test-slave-kghz"
server_key: "40d941b9b4a974f94dff22aeb1c9262b"
server_key_iv: "e21d070357a87c2da60def31cd3abfd5"
server_key_version: "encryptionkey@0"
I20260925 04:44:28.342474 3321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790311449873301-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790311449873301-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790311449873301-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:44:28.354157 3321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:44:28.357241 3321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:44:28.357438 3321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:44:28.364600 3321 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:46123
I20260925 04:44:28.364929 3388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:46123 every 8 connection(s)
I20260925 04:44:28.365602 3321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
I20260925 04:44:28.368450 3389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:44:28.372139 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 3321
I20260925 04:44:28.372275 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790311449873301-393-0/minicluster-data/master-0/wal/instance
I20260925 04:44:28.372558 393 external_mini_cluster.cc:1428] Setting key 6af36b939e835ed367d508849be30c01
I20260925 04:44:28.374192 3389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f37f800b8494dc3992a62c504b30076: Bootstrap starting.
I20260925 04:44:28.376668 3389 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f37f800b8494dc3992a62c504b30076: Neither blocks nor log segments found. Creating new log.
I20260925 04:44:28.377406 3389 log.cc:824] T 00000000000000000000000000000000 P 8f37f800b8494dc3992a62c504b30076: Log is configured to *not* fsync() on all Append() calls
I20260925 04:44:28.379534 3389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f37f800b8494dc3992a62c504b30076: No bootstrap required, opened a new log
Sep 25 04:44:28 dist-test-slave-kghz krb5kdc[3308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:44:28.384159 3389 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f37f800b8494dc3992a62c504b30076 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f37f800b8494dc3992a62c504b30076" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46123 } }
I20260925 04:44:28.384456 3389 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f37f800b8494dc3992a62c504b30076 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:44:28.384531 3389 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f37f800b8494dc3992a62c504b30076 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f37f800b8494dc3992a62c504b30076, State: Initialized, Role: FOLLOWER
I20260925 04:44:28.385031 3389 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f37f800b8494dc3992a62c504b30076 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f37f800b8494dc3992a62c504b30076" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46123 } }
I20260925 04:44:28.385205 3389 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f37f800b8494dc3992a62c504b30076 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:44:28.385272 3389 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f37f800b8494dc3992a62c504b30076 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:44:28.385372 3389 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f37f800b8494dc3992a62c504b30076 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:44:28.386341 3389 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f37f800b8494dc3992a62c504b30076 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f37f800b8494dc3992a62c504b30076" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46123 } }
I20260925 04:44:28.386704 3389 leader_election.cc:302] T 00000000000000000000000000000000 P 8f37f800b8494dc3992a62c504b30076 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f37f800b8494dc3992a62c504b30076; no voters:
I20260925 04:44:28.387007 3389 leader_election.cc:288] T 00000000000000000000000000000000 P 8f37f800b8494dc3992a62c504b30076 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:44:28.387132 3394 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f37f800b8494dc3992a62c504b30076 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:44:28.387333 3394 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f37f800b8494dc3992a62c504b30076 [term 1 LEADER]: Becoming Leader. State: Replica: 8f37f800b8494dc3992a62c504b30076, State: Running, Role: LEADER
I20260925 04:44:28.387529 3392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:28.373856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59890 (local address 127.0.98.126:46123)
0925 04:44:28.374338 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:28.374363 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:28.374412 (+ 49us) server_negotiation.cc:407] Connection header received
0925 04:44:28.375031 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:28.375058 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:28.375380 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:28.375658 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:28.376541 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:28.377659 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:28.378734 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:28.379201 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:28.379326 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:28.382088 (+ 2762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:28.382095 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:28.382113 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:28.382148 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:28.384253 (+ 2105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:28.384779 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:28.384785 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:28.384790 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:28.384933 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:28.385208 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:28.385215 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:28.385217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:28.385708 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:28.385922 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:28.386204 (+ 282us) server_negotiation.cc:299] Negotiation successful
0925 04:44:28.386567 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":124,"threads_started":1}
I20260925 04:44:28.387630 3394 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f37f800b8494dc3992a62c504b30076 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f37f800b8494dc3992a62c504b30076" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46123 } }
I20260925 04:44:28.388216 3389 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f37f800b8494dc3992a62c504b30076 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:44:28.388808 3396 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f37f800b8494dc3992a62c504b30076 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f37f800b8494dc3992a62c504b30076. Latest consensus state: current_term: 1 leader_uuid: "8f37f800b8494dc3992a62c504b30076" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f37f800b8494dc3992a62c504b30076" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46123 } } }
I20260925 04:44:28.388944 3396 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f37f800b8494dc3992a62c504b30076 [sys.catalog]: This master's current role is: LEADER
I20260925 04:44:28.389289 3400 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:44:28.390632 3395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f37f800b8494dc3992a62c504b30076 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f37f800b8494dc3992a62c504b30076" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f37f800b8494dc3992a62c504b30076" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46123 } } }
I20260925 04:44:28.390738 3395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f37f800b8494dc3992a62c504b30076 [sys.catalog]: This master's current role is: LEADER
I20260925 04:44:28.390810 393 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:44:28.392647 3400 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:44:28.399245 3400 catalog_manager.cc:1347] Generated new cluster ID: dd4460cb900243ca8c045b8a80ccaa86
I20260925 04:44:28.399317 3400 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:44:28.409725 3392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:28.404873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59904 (local address 127.0.98.126:46123)
0925 04:44:28.404999 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:28.405003 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:28.405252 (+ 249us) server_negotiation.cc:407] Connection header received
0925 04:44:28.405413 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:28.405416 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:28.405469 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:28.405558 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:28.406059 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:28.406683 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:28.407520 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:28.407760 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:28.407873 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:28.408752 (+ 879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:28.408756 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:28.408758 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:44:28.409259 (+ 501us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:28.409287 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:44:28.409303 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:28.409406 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:28.409494 (+ 88us) server_negotiation.cc:299] Negotiation successful
0925 04:44:28.409554 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 04:44:28.411564 3400 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:44:28.412485 3400 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:44:28.420677 3400 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f37f800b8494dc3992a62c504b30076: Generated new TSK 0
I20260925 04:44:28.421465 3400 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:44:28.426604 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 3321
2026-09-25T04:44:28Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1456 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:44:28 dist-test-slave-kghz krb5kdc[3422](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:44:28 dist-test-slave-kghz krb5kdc[3422](info): set up 2 sockets
Sep 25 04:44:28 dist-test-slave-kghz krb5kdc[3422](info): commencing operation
krb5kdc: starting...
W20260925 04:44:29.478444 393 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/krb5kdc/test-user.keytab.
Sep 25 04:44:29 dist-test-slave-kghz krb5kdc[3422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311469, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:44:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:44:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260925 04:44:29.610800 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:40167
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:37535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:40167
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:44:29.726861 3435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:44:29.727205 3435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:44:29.727285 3435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:44:29.731446 3435 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:44:29.731541 3435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:44:29.731576 3435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:44:29.731614 3435 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:44:29.731669 3435 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:44:29.737174 3435 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:37535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:40167
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:40167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:44:29.738436 3435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:44:29.739511 3435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:44:29.748425 3440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:29.748497 3441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:29.748485 3435 server_base.cc:1034] running on GCE node
W20260925 04:44:29.748430 3443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:29.749308 3435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:44:29.750552 3435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:44:29.751745 3435 hybrid_clock.cc:648] HybridClock initialized: now 1790311469751730 us; error 33 us; skew 500 ppm
Sep 25 04:44:29 dist-test-slave-kghz krb5kdc[3422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311469, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:44:29.754688 3435 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260925 04:44:29.755688 3435 webserver.cc:460] Webserver started at http://127.0.98.126:40547/ using document root <none> and password file <none>
I20260925 04:44:29.756326 3435 fs_manager.cc:359] Metadata directory not provided
I20260925 04:44:29.756390 3435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:44:29.756560 3435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:44:29.758823 3435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/minicluster-data/master-0/data/instance:
uuid: "bf1f6ea790e148bfadd4ef17f2d47bc9"
format_stamp: "Formatted at 2026-09-25 04:44:29 on dist-test-slave-kghz"
server_key: "8ce7c0cef9e1fda0622df7d8c619f319"
server_key_iv: "061c0047b4bdcc2e93bfb1d4812c7514"
server_key_version: "encryptionkey@0"
I20260925 04:44:29.759361 3435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/minicluster-data/master-0/wal/instance:
uuid: "bf1f6ea790e148bfadd4ef17f2d47bc9"
format_stamp: "Formatted at 2026-09-25 04:44:29 on dist-test-slave-kghz"
server_key: "8ce7c0cef9e1fda0622df7d8c619f319"
server_key_iv: "061c0047b4bdcc2e93bfb1d4812c7514"
server_key_version: "encryptionkey@0"
I20260925 04:44:29.763197 3435 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 04:44:29.765651 3450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:29.767055 3435 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:44:29.767201 3435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/minicluster-data/master-0/wal
uuid: "bf1f6ea790e148bfadd4ef17f2d47bc9"
format_stamp: "Formatted at 2026-09-25 04:44:29 on dist-test-slave-kghz"
server_key: "8ce7c0cef9e1fda0622df7d8c619f319"
server_key_iv: "061c0047b4bdcc2e93bfb1d4812c7514"
server_key_version: "encryptionkey@0"
I20260925 04:44:29.767318 3435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:44:29.793859 3435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:44:29.797379 3435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:44:29.797600 3435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:44:29.805006 3435 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:40167
I20260925 04:44:29.805303 3502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:40167 every 8 connection(s)
I20260925 04:44:29.806042 3435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
I20260925 04:44:29.807928 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 3435
I20260925 04:44:29.808071 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/minicluster-data/master-0/wal/instance
I20260925 04:44:29.808400 393 external_mini_cluster.cc:1428] Setting key a6cdeae4d3cbd78a4807ddf2ec33d933
I20260925 04:44:29.809724 3503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:44:29.816188 3503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf1f6ea790e148bfadd4ef17f2d47bc9: Bootstrap starting.
Sep 25 04:44:29 dist-test-slave-kghz krb5kdc[3422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311469, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:44:29.818718 3503 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf1f6ea790e148bfadd4ef17f2d47bc9: Neither blocks nor log segments found. Creating new log.
I20260925 04:44:29.819494 3503 log.cc:824] T 00000000000000000000000000000000 P bf1f6ea790e148bfadd4ef17f2d47bc9: Log is configured to *not* fsync() on all Append() calls
I20260925 04:44:29.821733 3503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf1f6ea790e148bfadd4ef17f2d47bc9: No bootstrap required, opened a new log
I20260925 04:44:29.823771 3506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:29.809982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52758 (local address 127.0.98.126:40167)
0925 04:44:29.810543 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:29.810560 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:29.810610 (+ 50us) server_negotiation.cc:407] Connection header received
0925 04:44:29.811333 (+ 723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:29.811359 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:29.811727 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:29.812162 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:29.812998 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:29.814012 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:29.814994 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:29.815494 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:29.815683 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:29.818326 (+ 2643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:29.818331 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:29.818347 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:29.818379 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:29.820433 (+ 2054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:29.820998 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:29.821003 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:29.821009 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:29.821127 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:29.821437 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:29.821441 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:29.821443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:29.821942 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:29.822144 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:29.822492 (+ 348us) server_negotiation.cc:299] Negotiation successful
0925 04:44:29.822806 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":199,"threads_started":1}
I20260925 04:44:29.825476 3503 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf1f6ea790e148bfadd4ef17f2d47bc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf1f6ea790e148bfadd4ef17f2d47bc9" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 40167 } }
I20260925 04:44:29.825752 3503 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf1f6ea790e148bfadd4ef17f2d47bc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:44:29.825842 3503 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf1f6ea790e148bfadd4ef17f2d47bc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf1f6ea790e148bfadd4ef17f2d47bc9, State: Initialized, Role: FOLLOWER
I20260925 04:44:29.826438 3503 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf1f6ea790e148bfadd4ef17f2d47bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf1f6ea790e148bfadd4ef17f2d47bc9" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 40167 } }
I20260925 04:44:29.826623 3503 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf1f6ea790e148bfadd4ef17f2d47bc9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:44:29.826691 3503 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf1f6ea790e148bfadd4ef17f2d47bc9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:44:29.826789 3503 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf1f6ea790e148bfadd4ef17f2d47bc9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:44:29.827782 3503 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf1f6ea790e148bfadd4ef17f2d47bc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf1f6ea790e148bfadd4ef17f2d47bc9" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 40167 } }
I20260925 04:44:29.828140 3503 leader_election.cc:302] T 00000000000000000000000000000000 P bf1f6ea790e148bfadd4ef17f2d47bc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf1f6ea790e148bfadd4ef17f2d47bc9; no voters:
I20260925 04:44:29.828485 3503 leader_election.cc:288] T 00000000000000000000000000000000 P bf1f6ea790e148bfadd4ef17f2d47bc9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:44:29.828632 3508 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf1f6ea790e148bfadd4ef17f2d47bc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:44:29.828896 3508 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf1f6ea790e148bfadd4ef17f2d47bc9 [term 1 LEADER]: Becoming Leader. State: Replica: bf1f6ea790e148bfadd4ef17f2d47bc9, State: Running, Role: LEADER
I20260925 04:44:29.829267 3508 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf1f6ea790e148bfadd4ef17f2d47bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf1f6ea790e148bfadd4ef17f2d47bc9" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 40167 } }
I20260925 04:44:29.829808 3503 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf1f6ea790e148bfadd4ef17f2d47bc9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:44:29.831166 3510 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf1f6ea790e148bfadd4ef17f2d47bc9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf1f6ea790e148bfadd4ef17f2d47bc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf1f6ea790e148bfadd4ef17f2d47bc9" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 40167 } } }
I20260925 04:44:29.831166 3509 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf1f6ea790e148bfadd4ef17f2d47bc9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf1f6ea790e148bfadd4ef17f2d47bc9. Latest consensus state: current_term: 1 leader_uuid: "bf1f6ea790e148bfadd4ef17f2d47bc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf1f6ea790e148bfadd4ef17f2d47bc9" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 40167 } } }
I20260925 04:44:29.831332 3509 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf1f6ea790e148bfadd4ef17f2d47bc9 [sys.catalog]: This master's current role is: LEADER
I20260925 04:44:29.831332 3510 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf1f6ea790e148bfadd4ef17f2d47bc9 [sys.catalog]: This master's current role is: LEADER
I20260925 04:44:29.831772 3519 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:44:29.835361 3519 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:44:29.841162 3519 catalog_manager.cc:1347] Generated new cluster ID: e34618096f83467f9c7f0ac8bf7bf6fb
I20260925 04:44:29.841259 3519 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:44:29.852972 3519 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:44:29.854363 3519 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:44:29.863507 3519 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf1f6ea790e148bfadd4ef17f2d47bc9: Generated new TSK 0
I20260925 04:44:29.864287 3519 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/krb5kdc/kudu_127.0.98.65.keytab.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/krb5kdc/kudu_127.0.98.65.keytab.
I20260925 04:44:29.914111 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:0
--local_ip_for_outbound_sockets=127.0.98.65
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:40167
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:37535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:44:30.033504 3529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:44:30.033840 3529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:44:30.033918 3529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:44:30.038177 3529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:44:30.038324 3529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:44:30.038424 3529 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260925 04:44:30.044632 3529 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:37535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:40167
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3529
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:44:30.045847 3529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:44:30.046991 3529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:44:30.058382 3534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:30.058382 3535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:30.058398 3537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:30.058784 3529 server_base.cc:1034] running on GCE node
I20260925 04:44:30.059237 3529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:44:30.059885 3529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:44:30.061107 3529 hybrid_clock.cc:648] HybridClock initialized: now 1790311470061087 us; error 32 us; skew 500 ppm
Sep 25 04:44:30 dist-test-slave-kghz krb5kdc[3422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311470, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:44:30.064208 3529 init.cc:377] Logged in from keytab as kudu/127.0.98.65@KRBTEST.COM (short username kudu)
I20260925 04:44:30.065177 3529 webserver.cc:460] Webserver started at http://127.0.98.65:40251/ using document root <none> and password file <none>
I20260925 04:44:30.065771 3529 fs_manager.cc:359] Metadata directory not provided
I20260925 04:44:30.065865 3529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:44:30.066115 3529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:44:30.068380 3529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-0/data/instance:
uuid: "45a651b1c26a41829d52e9218babaf5d"
format_stamp: "Formatted at 2026-09-25 04:44:30 on dist-test-slave-kghz"
server_key: "6a8a0409de5c2f1d9602b5848fa1d32f"
server_key_iv: "3e371928739fa9a0cf4484bb210a7362"
server_key_version: "encryptionkey@0"
I20260925 04:44:30.069005 3529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-0/wal/instance:
uuid: "45a651b1c26a41829d52e9218babaf5d"
format_stamp: "Formatted at 2026-09-25 04:44:30 on dist-test-slave-kghz"
server_key: "6a8a0409de5c2f1d9602b5848fa1d32f"
server_key_iv: "3e371928739fa9a0cf4484bb210a7362"
server_key_version: "encryptionkey@0"
I20260925 04:44:30.073189 3529 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260925 04:44:30.075997 3544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:30.077601 3529 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 04:44:30.077782 3529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-0/wal
uuid: "45a651b1c26a41829d52e9218babaf5d"
format_stamp: "Formatted at 2026-09-25 04:44:30 on dist-test-slave-kghz"
server_key: "6a8a0409de5c2f1d9602b5848fa1d32f"
server_key_iv: "3e371928739fa9a0cf4484bb210a7362"
server_key_version: "encryptionkey@0"
I20260925 04:44:30.077929 3529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:44:30.120349 3529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:44:30.124287 3529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:44:30.124541 3529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:44:30.125272 3529 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:44:30.126304 3529 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:44:30.126384 3529 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:30.126456 3529 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:44:30.126504 3529 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:30.137329 3529 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:45751
I20260925 04:44:30.137408 3657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:45751 every 8 connection(s)
I20260925 04:44:30.138396 3529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
I20260925 04:44:30.142071 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 3529
I20260925 04:44:30.142181 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-0/wal/instance
I20260925 04:44:30.142505 393 external_mini_cluster.cc:1428] Setting key 40a02e23f4760537bc289faea58bf905
Sep 25 04:44:30 dist-test-slave-kghz krb5kdc[3422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311470, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.66@KRBTEST.COM" created.
I20260925 04:44:30.155539 3506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:30.140681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:60749 (local address 127.0.98.126:40167)
0925 04:44:30.141066 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:30.141070 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:30.141790 (+ 720us) server_negotiation.cc:407] Connection header received
0925 04:44:30.143462 (+ 1672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:30.143465 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:30.143519 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:30.143601 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:30.144901 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:30.145570 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:30.147026 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:30.147482 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:30.147617 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:30.150881 (+ 3264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:30.150886 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:30.150890 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:30.150926 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:30.152790 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:30.153442 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:30.153513 (+ 71us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:30.153516 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:30.153616 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:30.154050 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:30.154053 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:30.154054 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:30.154367 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:30.154457 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:30.155141 (+ 684us) server_negotiation.cc:299] Negotiation successful
0925 04:44:30.155350 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240}
I20260925 04:44:30.156526 3660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:30.141081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:40167 (local address 127.0.98.65:60749)
0925 04:44:30.141601 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:30.141649 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:30.143220 (+ 1571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:30.143785 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:30.143793 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:30.144177 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:30.144711 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:30.144725 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:30.145759 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:30.145763 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:30.146872 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:30.146883 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:30.147668 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:30.147671 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:30.147933 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:30.148880 (+ 947us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:30.148898 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:30.150653 (+ 1755us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:44:30.152930 (+ 2277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:30.152936 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:30.152952 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:30.153279 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:30.153707 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:30.153710 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:30.153711 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:30.153874 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:30.154485 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:44:30.154519 (+ 34us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:44:30.154852 (+ 333us) client_negotiation.cc:769] Sending connection context
0925 04:44:30.155042 (+ 190us) client_negotiation.cc:240] Negotiation successful
0925 04:44:30.155387 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":134,"threads_started":1}
I20260925 04:44:30.157974 3658 heartbeater.cc:347] Connected to a master server at 127.0.98.126:40167
I20260925 04:44:30.158295 3658 heartbeater.cc:464] Registering TS with master...
I20260925 04:44:30.158879 3658 heartbeater.cc:511] Master 127.0.98.126:40167 requested a full tablet report, sending...
I20260925 04:44:30.160578 3467 ts_manager.cc:194] Registered new tserver with Master: 45a651b1c26a41829d52e9218babaf5d (127.0.98.65:45751)
I20260925 04:44:30.161818 3467 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.65@KRBTEST.COM'} at 127.0.98.65:60749
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/krb5kdc/kudu_127.0.98.66.keytab.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/krb5kdc/kudu_127.0.98.66.keytab.
I20260925 04:44:30.175638 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:0
--local_ip_for_outbound_sockets=127.0.98.66
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:40167
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:37535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:44:30.298396 3663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:44:30.298758 3663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:44:30.298808 3663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:44:30.303413 3663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:44:30.303512 3663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:44:30.303589 3663 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260925 04:44:30.309413 3663 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:37535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:40167
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3663
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:44:30.310725 3663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:44:30.312086 3663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:44:30.323336 3669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:30.323361 3671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:30.323342 3668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:30.323855 3663 server_base.cc:1034] running on GCE node
I20260925 04:44:30.324363 3663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:44:30.325158 3663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:44:30.326365 3663 hybrid_clock.cc:648] HybridClock initialized: now 1790311470326327 us; error 55 us; skew 500 ppm
Sep 25 04:44:30 dist-test-slave-kghz krb5kdc[3422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311470, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:44:30.330639 3663 init.cc:377] Logged in from keytab as kudu/127.0.98.66@KRBTEST.COM (short username kudu)
I20260925 04:44:30.331936 3663 webserver.cc:460] Webserver started at http://127.0.98.66:38479/ using document root <none> and password file <none>
I20260925 04:44:30.332604 3663 fs_manager.cc:359] Metadata directory not provided
I20260925 04:44:30.332676 3663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:44:30.332927 3663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:44:30.335615 3663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-1/data/instance:
uuid: "f464b476f3d04612a53fd45d06434c97"
format_stamp: "Formatted at 2026-09-25 04:44:30 on dist-test-slave-kghz"
server_key: "59a489165d75ad150ec3d61a1ebad085"
server_key_iv: "a69714d48ec3c4864bf63b6c477a38de"
server_key_version: "encryptionkey@0"
I20260925 04:44:30.336259 3663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-1/wal/instance:
uuid: "f464b476f3d04612a53fd45d06434c97"
format_stamp: "Formatted at 2026-09-25 04:44:30 on dist-test-slave-kghz"
server_key: "59a489165d75ad150ec3d61a1ebad085"
server_key_iv: "a69714d48ec3c4864bf63b6c477a38de"
server_key_version: "encryptionkey@0"
I20260925 04:44:30.340984 3663 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 04:44:30.344388 3678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:30.346134 3663 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 04:44:30.346376 3663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-1/wal
uuid: "f464b476f3d04612a53fd45d06434c97"
format_stamp: "Formatted at 2026-09-25 04:44:30 on dist-test-slave-kghz"
server_key: "59a489165d75ad150ec3d61a1ebad085"
server_key_iv: "a69714d48ec3c4864bf63b6c477a38de"
server_key_version: "encryptionkey@0"
I20260925 04:44:30.346526 3663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:44:30.369521 3663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:44:30.373397 3663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:44:30.373632 3663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:44:30.374240 3663 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:44:30.375375 3663 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:44:30.375428 3663 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:30.375483 3663 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:44:30.375501 3663 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:30.385581 3663 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:41973
I20260925 04:44:30.385591 3791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:41973 every 8 connection(s)
I20260925 04:44:30.386641 3663 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
I20260925 04:44:30.394132 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 3663
I20260925 04:44:30.394243 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-1/wal/instance
I20260925 04:44:30.394589 393 external_mini_cluster.cc:1428] Setting key 738ea33c775f873f24e9fc303490faaf
Sep 25 04:44:30 dist-test-slave-kghz krb5kdc[3422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311470, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:44:30.402717 3506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:30.388690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:41441 (local address 127.0.98.126:40167)
0925 04:44:30.388839 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:30.388843 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:30.389785 (+ 942us) server_negotiation.cc:407] Connection header received
0925 04:44:30.391321 (+ 1536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:30.391324 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:30.391376 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:30.391467 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:30.392847 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:30.393439 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:30.394862 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:30.395239 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:30.395362 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:30.398343 (+ 2981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:30.398349 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:30.398353 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:30.398389 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:30.400250 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:30.400897 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:30.400904 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:30.400909 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:30.401005 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:30.401354 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:30.401359 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:30.401363 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:30.401655 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:30.401775 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:30.402349 (+ 574us) server_negotiation.cc:299] Negotiation successful
0925 04:44:30.402506 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 04:44:30.403652 3794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:30.389007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:40167 (local address 127.0.98.66:41441)
0925 04:44:30.389593 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:30.389659 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:30.391074 (+ 1415us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:30.391707 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:30.391715 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:30.392149 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:30.392672 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:30.392687 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:30.393591 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:30.393594 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:30.394716 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:30.394729 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:30.395406 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:30.395409 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:30.395613 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:30.396298 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:30.396316 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:30.398113 (+ 1797us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:44:30.400410 (+ 2297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:30.400419 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:30.400441 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:30.400783 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:30.401105 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:30.401107 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:30.401108 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:30.401249 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:30.401786 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:44:30.401792 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:44:30.402058 (+ 266us) client_negotiation.cc:769] Sending connection context
0925 04:44:30.402240 (+ 182us) client_negotiation.cc:240] Negotiation successful
0925 04:44:30.402650 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":133,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.67@KRBTEST.COM; defaulting to no policy
I20260925 04:44:30.404979 3792 heartbeater.cc:347] Connected to a master server at 127.0.98.126:40167
I20260925 04:44:30.405315 3792 heartbeater.cc:464] Registering TS with master...
I20260925 04:44:30.405879 3792 heartbeater.cc:511] Master 127.0.98.126:40167 requested a full tablet report, sending...
I20260925 04:44:30.407202 3467 ts_manager.cc:194] Registered new tserver with Master: f464b476f3d04612a53fd45d06434c97 (127.0.98.66:41973)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.67@KRBTEST.COM" created.
I20260925 04:44:30.408154 3467 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.66@KRBTEST.COM'} at 127.0.98.66:41441
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/krb5kdc/kudu_127.0.98.67.keytab.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/krb5kdc/kudu_127.0.98.67.keytab.
I20260925 04:44:30.426157 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:0
--local_ip_for_outbound_sockets=127.0.98.67
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:40167
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:37535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:44:30.547341 3797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:44:30.547644 3797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:44:30.547727 3797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:44:30.552152 3797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:44:30.552266 3797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:44:30.552397 3797 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260925 04:44:30.558492 3797 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:37535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:40167
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:44:30.559713 3797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:44:30.560810 3797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:44:30.570716 3802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:30.570709 3805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:30.570716 3803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:30.571012 3797 server_base.cc:1034] running on GCE node
I20260925 04:44:30.571628 3797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:44:30.572324 3797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:44:30.573561 3797 hybrid_clock.cc:648] HybridClock initialized: now 1790311470573527 us; error 44 us; skew 500 ppm
Sep 25 04:44:30 dist-test-slave-kghz krb5kdc[3422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311470, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:44:30.576752 3797 init.cc:377] Logged in from keytab as kudu/127.0.98.67@KRBTEST.COM (short username kudu)
I20260925 04:44:30.577680 3797 webserver.cc:460] Webserver started at http://127.0.98.67:41443/ using document root <none> and password file <none>
I20260925 04:44:30.578312 3797 fs_manager.cc:359] Metadata directory not provided
I20260925 04:44:30.578389 3797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:44:30.578581 3797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:44:30.580731 3797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-2/data/instance:
uuid: "a849ea26476849cd97c0baa19bedcba1"
format_stamp: "Formatted at 2026-09-25 04:44:30 on dist-test-slave-kghz"
server_key: "61c31908afd4e4e7b07264592bbaa067"
server_key_iv: "2e57c2dbd73c4b88cb96410ed6c72e27"
server_key_version: "encryptionkey@0"
I20260925 04:44:30.581393 3797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-2/wal/instance:
uuid: "a849ea26476849cd97c0baa19bedcba1"
format_stamp: "Formatted at 2026-09-25 04:44:30 on dist-test-slave-kghz"
server_key: "61c31908afd4e4e7b07264592bbaa067"
server_key_iv: "2e57c2dbd73c4b88cb96410ed6c72e27"
server_key_version: "encryptionkey@0"
I20260925 04:44:30.585644 3797 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260925 04:44:30.588505 3812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:30.590113 3797 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 04:44:30.590492 3797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-2/wal
uuid: "a849ea26476849cd97c0baa19bedcba1"
format_stamp: "Formatted at 2026-09-25 04:44:30 on dist-test-slave-kghz"
server_key: "61c31908afd4e4e7b07264592bbaa067"
server_key_iv: "2e57c2dbd73c4b88cb96410ed6c72e27"
server_key_version: "encryptionkey@0"
I20260925 04:44:30.590636 3797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:44:30.610940 3797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:44:30.614575 3797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:44:30.614797 3797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:44:30.615489 3797 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:44:30.616348 3797 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:44:30.616420 3797 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:30.616492 3797 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:44:30.616539 3797 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:30.626015 3797 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:46229
I20260925 04:44:30.626034 3925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:46229 every 8 connection(s)
I20260925 04:44:30.627014 3797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
I20260925 04:44:30.633591 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 3797
I20260925 04:44:30.633744 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790311449873301-393-0/minicluster-data/ts-2/wal/instance
I20260925 04:44:30.634143 393 external_mini_cluster.cc:1428] Setting key 4be9332285fececd9a584e7301908a4d
Sep 25 04:44:30 dist-test-slave-kghz krb5kdc[3422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311470, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:44:30.643687 3506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:30.630487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:34935 (local address 127.0.98.126:40167)
0925 04:44:30.630607 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:30.630612 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:30.630635 (+ 23us) server_negotiation.cc:407] Connection header received
0925 04:44:30.632437 (+ 1802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:30.632440 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:30.632495 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:30.632578 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:30.633840 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:30.634654 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:30.635929 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:30.636208 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:30.636333 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:30.639224 (+ 2891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:30.639228 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:30.639230 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:30.639260 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:30.641099 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:30.641815 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:30.641819 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:30.641821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:30.641917 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:30.642382 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:30.642386 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:30.642388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:30.642608 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:30.642724 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:30.643331 (+ 607us) server_negotiation.cc:299] Negotiation successful
0925 04:44:30.643491 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 04:44:30.644624 3928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:30.629542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:40167 (local address 127.0.98.67:34935)
0925 04:44:30.630234 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:30.630325 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:30.632129 (+ 1804us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:30.632726 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:30.632733 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:30.633128 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:30.633650 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:30.633665 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:30.634808 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:30.634811 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:30.635768 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:30.635778 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:30.636377 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:30.636381 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:30.636589 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:30.637217 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:30.637234 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:30.639002 (+ 1768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:44:30.641277 (+ 2275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:30.641285 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:30.641307 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:30.641687 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:30.642034 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:30.642038 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:30.642040 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:30.642210 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:30.642739 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:44:30.642748 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:44:30.643059 (+ 311us) client_negotiation.cc:769] Sending connection context
0925 04:44:30.643269 (+ 210us) client_negotiation.cc:240] Negotiation successful
0925 04:44:30.643663 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":150,"threads_started":1}
I20260925 04:44:30.646008 3926 heartbeater.cc:347] Connected to a master server at 127.0.98.126:40167
I20260925 04:44:30.646368 3926 heartbeater.cc:464] Registering TS with master...
I20260925 04:44:30.646960 3926 heartbeater.cc:511] Master 127.0.98.126:40167 requested a full tablet report, sending...
I20260925 04:44:30.648115 3466 ts_manager.cc:194] Registered new tserver with Master: a849ea26476849cd97c0baa19bedcba1 (127.0.98.67:46229)
I20260925 04:44:30.648725 3466 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.67@KRBTEST.COM'} at 127.0.98.67:34935
I20260925 04:44:30.660626 393 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 04:44:30 dist-test-slave-kghz krb5kdc[3422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311470, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 25 04:44:30 dist-test-slave-kghz krb5kdc[3422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311470, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:44:30.687911 3506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:30.678304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52768 (local address 127.0.98.126:40167)
0925 04:44:30.678458 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:30.678462 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:30.678704 (+ 242us) server_negotiation.cc:407] Connection header received
0925 04:44:30.678840 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:30.678843 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:30.678892 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:30.678960 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:30.679847 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:30.680421 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:30.681316 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:30.681608 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:30.681708 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:30.684382 (+ 2674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:30.684385 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:30.684387 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:30.684420 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:30.686037 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:30.686610 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:30.686612 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:30.686614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:30.686695 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:30.686940 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:30.686942 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:30.686943 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:30.687219 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:30.687331 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:30.687644 (+ 313us) server_negotiation.cc:299] Negotiation successful
0925 04:44:30.687772 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260925 04:44:30.688356 3466 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:52768
I20260925 04:44:30.689399 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 3529
I20260925 04:44:30.695449 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 3663
I20260925 04:44:30.700959 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 3797
I20260925 04:44:30.707243 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 3435
2026-09-25T04:44:30Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2283 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:44:30 dist-test-slave-kghz krb5kdc[3938](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:44:30 dist-test-slave-kghz krb5kdc[3938](info): set up 2 sockets
Sep 25 04:44:30 dist-test-slave-kghz krb5kdc[3938](info): commencing operation
krb5kdc: starting...
W20260925 04:44:31.774775 393 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/krb5kdc/test-user.keytab.
Sep 25 04:44:31 dist-test-slave-kghz krb5kdc[3938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:44:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:44:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260925 04:44:31.910719 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:44875
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:40191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:44875
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:44:32.038195 3951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:44:32.038528 3951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:44:32.038573 3951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:44:32.042659 3951 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:44:32.042728 3951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:44:32.042747 3951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:44:32.042766 3951 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:44:32.042783 3951 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:44:32.048267 3951 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:40191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:44875
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:44875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3951
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:44:32.049373 3951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:44:32.050812 3951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:44:32.060863 3957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:32.060870 3956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:32.060931 3951 server_base.cc:1034] running on GCE node
W20260925 04:44:32.060870 3959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:32.061852 3951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:44:32.063185 3951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:44:32.064378 3951 hybrid_clock.cc:648] HybridClock initialized: now 1790311472064345 us; error 51 us; skew 500 ppm
Sep 25 04:44:32 dist-test-slave-kghz krb5kdc[3938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311472, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:44:32.067512 3951 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260925 04:44:32.068483 3951 webserver.cc:460] Webserver started at http://127.0.98.126:32989/ using document root <none> and password file <none>
I20260925 04:44:32.069024 3951 fs_manager.cc:359] Metadata directory not provided
I20260925 04:44:32.069087 3951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:44:32.069257 3951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:44:32.071434 3951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/minicluster-data/master-0/data/instance:
uuid: "4a171576efde45aa9d8c09b0d088afd7"
format_stamp: "Formatted at 2026-09-25 04:44:32 on dist-test-slave-kghz"
server_key: "263b1295619d1c752ec8a96c014b1f17"
server_key_iv: "0b974b7c9d8a9d3653616010b04cac59"
server_key_version: "encryptionkey@0"
I20260925 04:44:32.072022 3951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/minicluster-data/master-0/wal/instance:
uuid: "4a171576efde45aa9d8c09b0d088afd7"
format_stamp: "Formatted at 2026-09-25 04:44:32 on dist-test-slave-kghz"
server_key: "263b1295619d1c752ec8a96c014b1f17"
server_key_iv: "0b974b7c9d8a9d3653616010b04cac59"
server_key_version: "encryptionkey@0"
I20260925 04:44:32.076057 3951 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260925 04:44:32.078897 3966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:32.080470 3951 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:44:32.080892 3951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/minicluster-data/master-0/wal
uuid: "4a171576efde45aa9d8c09b0d088afd7"
format_stamp: "Formatted at 2026-09-25 04:44:32 on dist-test-slave-kghz"
server_key: "263b1295619d1c752ec8a96c014b1f17"
server_key_iv: "0b974b7c9d8a9d3653616010b04cac59"
server_key_version: "encryptionkey@0"
I20260925 04:44:32.081049 3951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:44:32.108269 3951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:44:32.112151 3951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:44:32.112485 3951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:44:32.120239 3951 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:44875
I20260925 04:44:32.120268 4018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:44875 every 8 connection(s)
I20260925 04:44:32.121271 3951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
I20260925 04:44:32.124325 4019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:44:32.128036 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 3951
I20260925 04:44:32.128165 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/minicluster-data/master-0/wal/instance
I20260925 04:44:32.128422 393 external_mini_cluster.cc:1428] Setting key 0c1138bf4bb7365f04e283462b61353d
I20260925 04:44:32.130893 4019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a171576efde45aa9d8c09b0d088afd7: Bootstrap starting.
I20260925 04:44:32.133293 4019 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a171576efde45aa9d8c09b0d088afd7: Neither blocks nor log segments found. Creating new log.
I20260925 04:44:32.134095 4019 log.cc:824] T 00000000000000000000000000000000 P 4a171576efde45aa9d8c09b0d088afd7: Log is configured to *not* fsync() on all Append() calls
I20260925 04:44:32.136379 4019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a171576efde45aa9d8c09b0d088afd7: No bootstrap required, opened a new log
Sep 25 04:44:32 dist-test-slave-kghz krb5kdc[3938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:44:32.140921 4019 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a171576efde45aa9d8c09b0d088afd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a171576efde45aa9d8c09b0d088afd7" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 44875 } }
I20260925 04:44:32.141232 4019 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a171576efde45aa9d8c09b0d088afd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:44:32.141304 4019 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a171576efde45aa9d8c09b0d088afd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a171576efde45aa9d8c09b0d088afd7, State: Initialized, Role: FOLLOWER
I20260925 04:44:32.141903 4019 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a171576efde45aa9d8c09b0d088afd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a171576efde45aa9d8c09b0d088afd7" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 44875 } }
I20260925 04:44:32.142118 4019 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a171576efde45aa9d8c09b0d088afd7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:44:32.142200 4019 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a171576efde45aa9d8c09b0d088afd7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:44:32.142334 4019 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a171576efde45aa9d8c09b0d088afd7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:44:32.144351 4022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:32.129998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43940 (local address 127.0.98.126:44875)
0925 04:44:32.130559 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:32.130623 (+ 64us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:32.130675 (+ 52us) server_negotiation.cc:407] Connection header received
0925 04:44:32.131419 (+ 744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:32.131443 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:32.131806 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:32.132169 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:32.133036 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:32.134095 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:32.135252 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:32.135738 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:32.135908 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:32.138918 (+ 3010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:32.138924 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:32.138971 (+ 47us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:32.139001 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:32.140985 (+ 1984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:32.141575 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:32.141581 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:32.141586 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:32.141709 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:32.142116 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:32.142121 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:32.142124 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:32.142726 (+ 602us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:32.142898 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:32.143214 (+ 316us) server_negotiation.cc:299] Negotiation successful
0925 04:44:32.143548 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":173,"threads_started":1}
I20260925 04:44:32.201740 4019 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a171576efde45aa9d8c09b0d088afd7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a171576efde45aa9d8c09b0d088afd7" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 44875 } }
I20260925 04:44:32.202318 4019 leader_election.cc:302] T 00000000000000000000000000000000 P 4a171576efde45aa9d8c09b0d088afd7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a171576efde45aa9d8c09b0d088afd7; no voters:
I20260925 04:44:32.202760 4019 leader_election.cc:288] T 00000000000000000000000000000000 P 4a171576efde45aa9d8c09b0d088afd7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:44:32.202946 4024 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a171576efde45aa9d8c09b0d088afd7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:44:32.203205 4024 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a171576efde45aa9d8c09b0d088afd7 [term 1 LEADER]: Becoming Leader. State: Replica: 4a171576efde45aa9d8c09b0d088afd7, State: Running, Role: LEADER
I20260925 04:44:32.203703 4024 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a171576efde45aa9d8c09b0d088afd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a171576efde45aa9d8c09b0d088afd7" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 44875 } }
I20260925 04:44:32.204200 4019 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a171576efde45aa9d8c09b0d088afd7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:44:32.205626 4025 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a171576efde45aa9d8c09b0d088afd7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a171576efde45aa9d8c09b0d088afd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a171576efde45aa9d8c09b0d088afd7" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 44875 } } }
I20260925 04:44:32.205783 4025 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a171576efde45aa9d8c09b0d088afd7 [sys.catalog]: This master's current role is: LEADER
I20260925 04:44:32.206285 4026 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a171576efde45aa9d8c09b0d088afd7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a171576efde45aa9d8c09b0d088afd7. Latest consensus state: current_term: 1 leader_uuid: "4a171576efde45aa9d8c09b0d088afd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a171576efde45aa9d8c09b0d088afd7" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 44875 } } }
I20260925 04:44:32.206442 4026 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a171576efde45aa9d8c09b0d088afd7 [sys.catalog]: This master's current role is: LEADER
W20260925 04:44:32.209674 4039 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4a171576efde45aa9d8c09b0d088afd7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 04:44:32.209775 4039 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 04:44:32.209841 4033 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:44:32.210629 4033 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:44:32.216179 4033 catalog_manager.cc:1347] Generated new cluster ID: fac5aeaa3fb943d2a0a72edf4012eae4
I20260925 04:44:32.216264 4033 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:44:32.230644 4033 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:44:32.231848 4033 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:44:32.236512 4033 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a171576efde45aa9d8c09b0d088afd7: Generated new TSK 0
I20260925 04:44:32.237429 4033 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/krb5kdc/kudu_127.0.98.65.keytab.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/krb5kdc/kudu_127.0.98.65.keytab.
W20260925 04:44:32.365736 393 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.65: real 0.124s user 0.001s sys 0.003s
I20260925 04:44:32.366269 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:0
--local_ip_for_outbound_sockets=127.0.98.65
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:44875
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:40191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:44:32.481657 4045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:44:32.481980 4045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:44:32.482033 4045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:44:32.486436 4045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:44:32.486526 4045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:44:32.486595 4045 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260925 04:44:32.492262 4045 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:40191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:44875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4045
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:44:32.493464 4045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:44:32.494648 4045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:44:32.505789 4050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:32.505810 4051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:32.505777 4053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:32.506184 4045 server_base.cc:1034] running on GCE node
I20260925 04:44:32.506742 4045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:44:32.507485 4045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:44:32.508672 4045 hybrid_clock.cc:648] HybridClock initialized: now 1790311472508673 us; error 47 us; skew 500 ppm
Sep 25 04:44:32 dist-test-slave-kghz krb5kdc[3938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311472, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:44:32.515184 4045 init.cc:377] Logged in from keytab as kudu/127.0.98.65@KRBTEST.COM (short username kudu)
I20260925 04:44:32.516175 4045 webserver.cc:460] Webserver started at http://127.0.98.65:43431/ using document root <none> and password file <none>
I20260925 04:44:32.516808 4045 fs_manager.cc:359] Metadata directory not provided
I20260925 04:44:32.516893 4045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:44:32.517086 4045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:44:32.519434 4045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/minicluster-data/ts-0/data/instance:
uuid: "e5ecdc5c04ab48f89e6cc14d1e4592f3"
format_stamp: "Formatted at 2026-09-25 04:44:32 on dist-test-slave-kghz"
server_key: "f78fde2d5fa32368d9ad3c44744c97a0"
server_key_iv: "a45e72f35c91d9effa5564d9363f056c"
server_key_version: "encryptionkey@0"
I20260925 04:44:32.520036 4045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/minicluster-data/ts-0/wal/instance:
uuid: "e5ecdc5c04ab48f89e6cc14d1e4592f3"
format_stamp: "Formatted at 2026-09-25 04:44:32 on dist-test-slave-kghz"
server_key: "f78fde2d5fa32368d9ad3c44744c97a0"
server_key_iv: "a45e72f35c91d9effa5564d9363f056c"
server_key_version: "encryptionkey@0"
I20260925 04:44:32.524237 4045 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 04:44:32.527081 4060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:32.528497 4045 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 04:44:32.528674 4045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/minicluster-data/ts-0/wal
uuid: "e5ecdc5c04ab48f89e6cc14d1e4592f3"
format_stamp: "Formatted at 2026-09-25 04:44:32 on dist-test-slave-kghz"
server_key: "f78fde2d5fa32368d9ad3c44744c97a0"
server_key_iv: "a45e72f35c91d9effa5564d9363f056c"
server_key_version: "encryptionkey@0"
I20260925 04:44:32.528818 4045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:44:32.543000 4045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:44:32.546707 4045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:44:32.546940 4045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:44:32.547590 4045 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:44:32.548462 4045 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:44:32.548539 4045 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:32.548609 4045 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:44:32.548662 4045 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:32.559885 4045 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:41367
I20260925 04:44:32.559954 4173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:41367 every 8 connection(s)
I20260925 04:44:32.561120 4045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
I20260925 04:44:32.563651 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 4045
I20260925 04:44:32.563895 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/minicluster-data/ts-0/wal/instance
I20260925 04:44:32.564415 393 external_mini_cluster.cc:1428] Setting key dda5f40775890942f387166e5e66bd8a
Sep 25 04:44:32 dist-test-slave-kghz krb5kdc[3938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311472, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.66@KRBTEST.COM" created.
I20260925 04:44:32.582814 4022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:32.563778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:43425 (local address 127.0.98.126:44875)
0925 04:44:32.563915 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:32.563922 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:32.565584 (+ 1662us) server_negotiation.cc:407] Connection header received
0925 04:44:32.567376 (+ 1792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:32.567380 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:32.567440 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:32.567519 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:32.569308 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:32.569858 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:32.571810 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:32.572283 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:32.572382 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:32.576220 (+ 3838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:32.576226 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:32.576229 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:32.576287 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:32.579847 (+ 3560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:32.580750 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:32.580756 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:32.580758 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:32.580867 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:32.581385 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:32.581389 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:32.581391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:32.581611 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:32.581732 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:32.582496 (+ 764us) server_negotiation.cc:299] Negotiation successful
0925 04:44:32.582637 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 04:44:32.584107 4176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:32.564353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:44875 (local address 127.0.98.65:43425)
0925 04:44:32.565218 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:32.565283 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:32.567117 (+ 1834us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:32.567715 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:32.567741 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:32.568293 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:32.569123 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:32.569157 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:32.570010 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:32.570015 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:32.571627 (+ 1612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:32.571639 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:32.572643 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:32.572650 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:32.572892 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:32.573582 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:32.573620 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:32.575938 (+ 2318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:44:32.580029 (+ 4091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:32.580039 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:32.580067 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:32.580563 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:32.580997 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:32.581000 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:32.581002 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:32.581229 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:32.581729 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:44:32.581737 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:44:32.582150 (+ 413us) client_negotiation.cc:769] Sending connection context
0925 04:44:32.582427 (+ 277us) client_negotiation.cc:240] Negotiation successful
0925 04:44:32.582807 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":389,"threads_started":1}
I20260925 04:44:32.585749 4174 heartbeater.cc:347] Connected to a master server at 127.0.98.126:44875
I20260925 04:44:32.586026 4174 heartbeater.cc:464] Registering TS with master...
I20260925 04:44:32.586752 4174 heartbeater.cc:511] Master 127.0.98.126:44875 requested a full tablet report, sending...
I20260925 04:44:32.588709 3980 ts_manager.cc:194] Registered new tserver with Master: e5ecdc5c04ab48f89e6cc14d1e4592f3 (127.0.98.65:41367)
I20260925 04:44:32.590353 3980 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.65@KRBTEST.COM'} at 127.0.98.65:43425
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/krb5kdc/kudu_127.0.98.66.keytab.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/krb5kdc/kudu_127.0.98.66.keytab.
I20260925 04:44:32.602210 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:0
--local_ip_for_outbound_sockets=127.0.98.66
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:44875
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:40191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:44:32.719170 4179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:44:32.719489 4179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:44:32.719573 4179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:44:32.723855 4179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:44:32.723958 4179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:44:32.724053 4179 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260925 04:44:32.729610 4179 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:40191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:44875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4179
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:44:32.730778 4179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:44:32.731856 4179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:44:32.741756 4185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:32.741746 4184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:32.741746 4187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:32.742053 4179 server_base.cc:1034] running on GCE node
I20260925 04:44:32.742630 4179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:44:32.743325 4179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:44:32.744513 4179 hybrid_clock.cc:648] HybridClock initialized: now 1790311472744509 us; error 32 us; skew 500 ppm
Sep 25 04:44:32 dist-test-slave-kghz krb5kdc[3938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311472, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:44:32.747344 4179 init.cc:377] Logged in from keytab as kudu/127.0.98.66@KRBTEST.COM (short username kudu)
I20260925 04:44:32.748296 4179 webserver.cc:460] Webserver started at http://127.0.98.66:37397/ using document root <none> and password file <none>
I20260925 04:44:32.748870 4179 fs_manager.cc:359] Metadata directory not provided
I20260925 04:44:32.748934 4179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:44:32.749146 4179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:44:32.751302 4179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/minicluster-data/ts-1/data/instance:
uuid: "5c13f3c5167149f5b8964a2e633c147d"
format_stamp: "Formatted at 2026-09-25 04:44:32 on dist-test-slave-kghz"
server_key: "079e847de13e75150e8832d4a05b6bcc"
server_key_iv: "7d71cecf39ee619c62cfc7627a46b2e4"
server_key_version: "encryptionkey@0"
I20260925 04:44:32.751896 4179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/minicluster-data/ts-1/wal/instance:
uuid: "5c13f3c5167149f5b8964a2e633c147d"
format_stamp: "Formatted at 2026-09-25 04:44:32 on dist-test-slave-kghz"
server_key: "079e847de13e75150e8832d4a05b6bcc"
server_key_iv: "7d71cecf39ee619c62cfc7627a46b2e4"
server_key_version: "encryptionkey@0"
I20260925 04:44:32.755921 4179 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 04:44:32.758565 4194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:32.759827 4179 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 04:44:32.759966 4179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/minicluster-data/ts-1/wal
uuid: "5c13f3c5167149f5b8964a2e633c147d"
format_stamp: "Formatted at 2026-09-25 04:44:32 on dist-test-slave-kghz"
server_key: "079e847de13e75150e8832d4a05b6bcc"
server_key_iv: "7d71cecf39ee619c62cfc7627a46b2e4"
server_key_version: "encryptionkey@0"
I20260925 04:44:32.760084 4179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:44:32.792380 4179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:44:32.796168 4179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:44:32.796411 4179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:44:32.797119 4179 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:44:32.798009 4179 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:44:32.798082 4179 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:32.798151 4179 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:44:32.798202 4179 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:32.808753 4179 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:43767
I20260925 04:44:32.808775 4307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:43767 every 8 connection(s)
I20260925 04:44:32.809774 4179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
Sep 25 04:44:32 dist-test-slave-kghz krb5kdc[3938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311472, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:44:32.819831 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 4179
I20260925 04:44:32.819952 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/minicluster-data/ts-1/wal/instance
I20260925 04:44:32.820263 393 external_mini_cluster.cc:1428] Setting key 2db4ae57cb145f3f24a218fe8a7141e6
I20260925 04:44:32.825179 4022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:32.811803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:49553 (local address 127.0.98.126:44875)
0925 04:44:32.811946 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:32.811950 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:32.812781 (+ 831us) server_negotiation.cc:407] Connection header received
0925 04:44:32.814284 (+ 1503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:32.814288 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:32.814347 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:32.814431 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:32.815661 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:32.816204 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:32.817400 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:32.817650 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:32.817766 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:32.820534 (+ 2768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:32.820538 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:32.820541 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:32.820574 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:32.822232 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:32.822972 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:32.822975 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:32.822976 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:32.823067 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:32.823723 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:32.823728 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:32.823731 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:32.824057 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:32.824171 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:32.824755 (+ 584us) server_negotiation.cc:299] Negotiation successful
0925 04:44:32.824906 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 04:44:32.825981 4310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:32.812141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:44875 (local address 127.0.98.66:49553)
0925 04:44:32.812614 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:32.812660 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:32.814024 (+ 1364us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:32.814610 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:32.814617 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:32.814997 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:32.815499 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:32.815514 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:32.816340 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:32.816344 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:32.817267 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:32.817276 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:32.817808 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:32.817812 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:32.818003 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:32.818604 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:32.818621 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:32.820333 (+ 1712us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:44:32.822390 (+ 2057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:32.822399 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:32.822420 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:32.822843 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:32.823335 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:32.823340 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:32.823342 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:32.823564 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:32.824185 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:44:32.824190 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:44:32.824481 (+ 291us) client_negotiation.cc:769] Sending connection context
0925 04:44:32.824669 (+ 188us) client_negotiation.cc:240] Negotiation successful
0925 04:44:32.824950 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":126,"threads_started":1}
I20260925 04:44:32.827283 4308 heartbeater.cc:347] Connected to a master server at 127.0.98.126:44875
I20260925 04:44:32.827541 4308 heartbeater.cc:464] Registering TS with master...
I20260925 04:44:32.828084 4308 heartbeater.cc:511] Master 127.0.98.126:44875 requested a full tablet report, sending...
I20260925 04:44:32.829265 3980 ts_manager.cc:194] Registered new tserver with Master: 5c13f3c5167149f5b8964a2e633c147d (127.0.98.66:43767)
I20260925 04:44:32.829908 3980 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.66@KRBTEST.COM'} at 127.0.98.66:49553
WARNING: no policy specified for kudu/127.0.98.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/krb5kdc/kudu_127.0.98.67.keytab.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/krb5kdc/kudu_127.0.98.67.keytab.
I20260925 04:44:32.850638 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:0
--local_ip_for_outbound_sockets=127.0.98.67
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:44875
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:40191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:44:32.969614 4313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:44:32.969962 4313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:44:32.970052 4313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:44:32.974292 4313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:44:32.974403 4313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:44:32.974501 4313 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260925 04:44:32.980118 4313 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:40191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:44875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:44:32.981300 4313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:44:32.982445 4313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:44:32.992877 4319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:32.992928 4318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:32.992944 4313 server_base.cc:1034] running on GCE node
W20260925 04:44:32.993060 4321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:32.993625 4313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:44:32.994433 4313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:44:32.995654 4313 hybrid_clock.cc:648] HybridClock initialized: now 1790311472995629 us; error 45 us; skew 500 ppm
Sep 25 04:44:32 dist-test-slave-kghz krb5kdc[3938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311472, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:44:32.998745 4313 init.cc:377] Logged in from keytab as kudu/127.0.98.67@KRBTEST.COM (short username kudu)
I20260925 04:44:32.999687 4313 webserver.cc:460] Webserver started at http://127.0.98.67:45573/ using document root <none> and password file <none>
I20260925 04:44:33.000281 4313 fs_manager.cc:359] Metadata directory not provided
I20260925 04:44:33.000344 4313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:44:33.000576 4313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:44:33.003051 4313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/minicluster-data/ts-2/data/instance:
uuid: "6481683c6205444f82283ba4707857e2"
format_stamp: "Formatted at 2026-09-25 04:44:32 on dist-test-slave-kghz"
server_key: "4d1c312ae7b99885ad93d0aea79c65ba"
server_key_iv: "254fb848de47efcfb0d497e44bcab5e1"
server_key_version: "encryptionkey@0"
I20260925 04:44:33.003706 4313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/minicluster-data/ts-2/wal/instance:
uuid: "6481683c6205444f82283ba4707857e2"
format_stamp: "Formatted at 2026-09-25 04:44:32 on dist-test-slave-kghz"
server_key: "4d1c312ae7b99885ad93d0aea79c65ba"
server_key_iv: "254fb848de47efcfb0d497e44bcab5e1"
server_key_version: "encryptionkey@0"
I20260925 04:44:33.008420 4313 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 04:44:33.011418 4328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:33.013165 4313 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:44:33.013340 4313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/minicluster-data/ts-2/wal
uuid: "6481683c6205444f82283ba4707857e2"
format_stamp: "Formatted at 2026-09-25 04:44:32 on dist-test-slave-kghz"
server_key: "4d1c312ae7b99885ad93d0aea79c65ba"
server_key_iv: "254fb848de47efcfb0d497e44bcab5e1"
server_key_version: "encryptionkey@0"
I20260925 04:44:33.013477 4313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:44:33.032483 4313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:44:33.036458 4313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:44:33.036677 4313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:44:33.037397 4313 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:44:33.038347 4313 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:44:33.038396 4313 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:33.038439 4313 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:44:33.038491 4313 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:33.048774 4313 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:35661
I20260925 04:44:33.048794 4441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:35661 every 8 connection(s)
I20260925 04:44:33.049732 4313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
I20260925 04:44:33.057718 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 4313
I20260925 04:44:33.057883 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790311449873301-393-0/minicluster-data/ts-2/wal/instance
I20260925 04:44:33.058240 393 external_mini_cluster.cc:1428] Setting key 67361b00cd93b2af87b9fa848db64f90
Sep 25 04:44:33 dist-test-slave-kghz krb5kdc[3938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311472, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:44:33.066538 4022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:33.051694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:59553 (local address 127.0.98.126:44875)
0925 04:44:33.051837 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:33.051840 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:33.052712 (+ 872us) server_negotiation.cc:407] Connection header received
0925 04:44:33.054707 (+ 1995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:33.054711 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:33.054782 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:33.054873 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:33.056923 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:33.057579 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:33.059169 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:33.059426 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:33.059565 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:33.062508 (+ 2943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:33.062513 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:33.062523 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:33.062559 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:33.064338 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:33.064985 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:33.064988 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:33.064990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:33.065079 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:33.065398 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:33.065416 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:33.065417 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:33.065614 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:33.065689 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:33.066235 (+ 546us) server_negotiation.cc:299] Negotiation successful
0925 04:44:33.066408 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 04:44:33.067415 4444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:33.052067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:44875 (local address 127.0.98.67:59553)
0925 04:44:33.052535 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:33.052581 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:33.054436 (+ 1855us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:33.055126 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:33.055143 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:33.055860 (+ 717us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:33.056737 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:33.056756 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:33.057728 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:33.057732 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:33.059014 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:33.059025 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:33.059603 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:33.059607 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:33.059821 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:33.060541 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:33.060562 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:33.062305 (+ 1743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:44:33.064479 (+ 2174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:33.064487 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:33.064505 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:33.064806 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:33.065153 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:33.065156 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:33.065157 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:33.065308 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:33.065701 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:44:33.065706 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:44:33.065968 (+ 262us) client_negotiation.cc:769] Sending connection context
0925 04:44:33.066164 (+ 196us) client_negotiation.cc:240] Negotiation successful
0925 04:44:33.066521 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":122,"threads_started":1}
I20260925 04:44:33.068696 4442 heartbeater.cc:347] Connected to a master server at 127.0.98.126:44875
I20260925 04:44:33.068957 4442 heartbeater.cc:464] Registering TS with master...
I20260925 04:44:33.069478 4442 heartbeater.cc:511] Master 127.0.98.126:44875 requested a full tablet report, sending...
I20260925 04:44:33.070674 3980 ts_manager.cc:194] Registered new tserver with Master: 6481683c6205444f82283ba4707857e2 (127.0.98.67:35661)
I20260925 04:44:33.071455 3980 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.67@KRBTEST.COM'} at 127.0.98.67:59553
I20260925 04:44:33.072307 393 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 04:44:33 dist-test-slave-kghz krb5kdc[3938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 04:44:33 dist-test-slave-kghz krb5kdc[3938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.65@KRBTEST.COM
I20260925 04:44:33.100016 4449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:33.089419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46708 (local address 127.0.98.65:41367)
0925 04:44:33.089708 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:33.089717 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:33.089744 (+ 27us) server_negotiation.cc:407] Connection header received
0925 04:44:33.089882 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:33.089890 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:33.090111 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:33.090308 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:33.091044 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:33.091835 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:33.092740 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:33.093142 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:33.093262 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:33.096192 (+ 2930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:33.096202 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:33.096217 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:33.096245 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:33.098005 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:33.098586 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:33.098592 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:33.098593 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:33.098682 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:33.098967 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:33.098970 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:33.098972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:33.099268 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:33.099507 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:33.099677 (+ 170us) server_negotiation.cc:299] Negotiation successful
0925 04:44:33.099841 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":63,"threads_started":1}
Sep 25 04:44:33 dist-test-slave-kghz krb5kdc[3938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:44:33.112886 4022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:33.103403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43942 (local address 127.0.98.126:44875)
0925 04:44:33.103556 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:33.103562 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:33.103662 (+ 100us) server_negotiation.cc:407] Connection header received
0925 04:44:33.103827 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:33.103830 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:33.103879 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:33.103965 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:33.104671 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:33.105316 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:33.106218 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:33.106637 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:33.106749 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:33.109341 (+ 2592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:33.109345 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:33.109347 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:33.109378 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:33.110979 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:33.111401 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:33.111403 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:33.111405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:33.111483 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:33.111811 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:33.111814 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:33.111815 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:33.112214 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:33.112336 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:33.112634 (+ 298us) server_negotiation.cc:299] Negotiation successful
0925 04:44:33.112747 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260925 04:44:33.113415 3980 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43942
I20260925 04:44:33.114212 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 4045
I20260925 04:44:33.120074 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 4179
I20260925 04:44:33.126868 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 4313
I20260925 04:44:33.132381 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 3951
2026-09-25T04:44:33Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2424 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:44:33 dist-test-slave-kghz krb5kdc[4454](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:44:33 dist-test-slave-kghz krb5kdc[4454](info): set up 2 sockets
Sep 25 04:44:33 dist-test-slave-kghz krb5kdc[4454](info): commencing operation
krb5kdc: starting...
W20260925 04:44:34.194574 393 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/krb5kdc/test-user.keytab.
Sep 25 04:44:34 dist-test-slave-kghz krb5kdc[4454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:44:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:44:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260925 04:44:34.325912 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:41901
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:39589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:41901
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:44:34.440761 4467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:44:34.441125 4467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:44:34.441206 4467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:44:34.445729 4467 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:44:34.445860 4467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:44:34.445900 4467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:44:34.445942 4467 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:44:34.445976 4467 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:44:34.451727 4467 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:39589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:41901
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:41901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4467
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:44:34.452879 4467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:44:34.453949 4467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:44:34.462891 4472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:34.463096 4473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:34.462891 4475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:34.462975 4467 server_base.cc:1034] running on GCE node
I20260925 04:44:34.463809 4467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:44:34.465027 4467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:44:34.466344 4467 hybrid_clock.cc:648] HybridClock initialized: now 1790311474466330 us; error 30 us; skew 500 ppm
Sep 25 04:44:34 dist-test-slave-kghz krb5kdc[4454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311474, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:44:34.474856 4467 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260925 04:44:34.475719 4467 fs_manager.cc:359] Metadata directory not provided
I20260925 04:44:34.475793 4467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:44:34.475970 4467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:44:34.478343 4467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/minicluster-data/master-0/data/instance:
uuid: "12a0987c0c964debb984f75c242e646c"
format_stamp: "Formatted at 2026-09-25 04:44:34 on dist-test-slave-kghz"
server_key: "0660e8406906eb54cbd094458b2ba435"
server_key_iv: "6056bb7ddd2b7a4022837c425d9fce96"
server_key_version: "encryptionkey@0"
I20260925 04:44:34.478937 4467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/minicluster-data/master-0/wal/instance:
uuid: "12a0987c0c964debb984f75c242e646c"
format_stamp: "Formatted at 2026-09-25 04:44:34 on dist-test-slave-kghz"
server_key: "0660e8406906eb54cbd094458b2ba435"
server_key_iv: "6056bb7ddd2b7a4022837c425d9fce96"
server_key_version: "encryptionkey@0"
I20260925 04:44:34.482925 4467 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 04:44:34.485682 4481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:34.487123 4467 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:44:34.487234 4467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/minicluster-data/master-0/wal
uuid: "12a0987c0c964debb984f75c242e646c"
format_stamp: "Formatted at 2026-09-25 04:44:34 on dist-test-slave-kghz"
server_key: "0660e8406906eb54cbd094458b2ba435"
server_key_iv: "6056bb7ddd2b7a4022837c425d9fce96"
server_key_version: "encryptionkey@0"
I20260925 04:44:34.487330 4467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:44:34.510099 4467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:44:34.513684 4467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:44:34.513870 4467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:44:34.521304 4467 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:41901
I20260925 04:44:34.521613 4533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:41901 every 8 connection(s)
I20260925 04:44:34.522038 4467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
I20260925 04:44:34.522756 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 4467
I20260925 04:44:34.522886 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/minicluster-data/master-0/wal/instance
I20260925 04:44:34.523150 393 external_mini_cluster.cc:1428] Setting key 2c4ac26a432cc17ee1fabe6fa1018e1f
I20260925 04:44:34.525481 4534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:44:34.532475 4534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12a0987c0c964debb984f75c242e646c: Bootstrap starting.
Sep 25 04:44:34 dist-test-slave-kghz krb5kdc[4454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:44:34.535382 4534 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12a0987c0c964debb984f75c242e646c: Neither blocks nor log segments found. Creating new log.
I20260925 04:44:34.536301 4534 log.cc:824] T 00000000000000000000000000000000 P 12a0987c0c964debb984f75c242e646c: Log is configured to *not* fsync() on all Append() calls
I20260925 04:44:34.538774 4534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12a0987c0c964debb984f75c242e646c: No bootstrap required, opened a new log
I20260925 04:44:34.539039 4537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:34.524730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35472 (local address 127.0.98.126:41901)
0925 04:44:34.525198 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:34.525217 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:34.525261 (+ 44us) server_negotiation.cc:407] Connection header received
0925 04:44:34.526129 (+ 868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:34.526152 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:34.526614 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:34.526996 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:34.527722 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:34.529261 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:34.530208 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:34.530949 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:34.531116 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:34.533707 (+ 2591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:34.533714 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:34.533730 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:34.533772 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:34.535898 (+ 2126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:34.536350 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:34.536356 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:34.536361 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:34.536499 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:34.536797 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:34.536800 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:34.536801 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:34.537290 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:34.537476 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:34.537753 (+ 277us) server_negotiation.cc:299] Negotiation successful
0925 04:44:34.538048 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":131,"threads_started":1}
I20260925 04:44:34.542793 4534 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12a0987c0c964debb984f75c242e646c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12a0987c0c964debb984f75c242e646c" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41901 } }
I20260925 04:44:34.543036 4534 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12a0987c0c964debb984f75c242e646c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:44:34.543072 4534 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12a0987c0c964debb984f75c242e646c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12a0987c0c964debb984f75c242e646c, State: Initialized, Role: FOLLOWER
I20260925 04:44:34.543450 4534 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12a0987c0c964debb984f75c242e646c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12a0987c0c964debb984f75c242e646c" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41901 } }
I20260925 04:44:34.543562 4534 raft_consensus.cc:388] T 00000000000000000000000000000000 P 12a0987c0c964debb984f75c242e646c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:44:34.543646 4534 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12a0987c0c964debb984f75c242e646c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:44:34.543747 4534 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12a0987c0c964debb984f75c242e646c [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:44:34.544862 4534 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12a0987c0c964debb984f75c242e646c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12a0987c0c964debb984f75c242e646c" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41901 } }
I20260925 04:44:34.545187 4534 leader_election.cc:302] T 00000000000000000000000000000000 P 12a0987c0c964debb984f75c242e646c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12a0987c0c964debb984f75c242e646c; no voters:
I20260925 04:44:34.545475 4534 leader_election.cc:288] T 00000000000000000000000000000000 P 12a0987c0c964debb984f75c242e646c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:44:34.545624 4539 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12a0987c0c964debb984f75c242e646c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:44:34.545877 4539 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12a0987c0c964debb984f75c242e646c [term 1 LEADER]: Becoming Leader. State: Replica: 12a0987c0c964debb984f75c242e646c, State: Running, Role: LEADER
I20260925 04:44:34.546221 4539 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12a0987c0c964debb984f75c242e646c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12a0987c0c964debb984f75c242e646c" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41901 } }
I20260925 04:44:34.546797 4534 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12a0987c0c964debb984f75c242e646c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:44:34.548146 4540 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12a0987c0c964debb984f75c242e646c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12a0987c0c964debb984f75c242e646c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12a0987c0c964debb984f75c242e646c" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41901 } } }
I20260925 04:44:34.548168 4541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12a0987c0c964debb984f75c242e646c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12a0987c0c964debb984f75c242e646c. Latest consensus state: current_term: 1 leader_uuid: "12a0987c0c964debb984f75c242e646c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12a0987c0c964debb984f75c242e646c" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41901 } } }
I20260925 04:44:34.548307 4541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12a0987c0c964debb984f75c242e646c [sys.catalog]: This master's current role is: LEADER
I20260925 04:44:34.548307 4540 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12a0987c0c964debb984f75c242e646c [sys.catalog]: This master's current role is: LEADER
I20260925 04:44:34.548753 4549 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:44:34.552333 4549 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:44:34.557688 4549 catalog_manager.cc:1347] Generated new cluster ID: 5ad2af8752304dc4a0a160b80f62e7f1
I20260925 04:44:34.557771 4549 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:44:34.572700 4549 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:44:34.573643 4549 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:44:34.581847 4549 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 12a0987c0c964debb984f75c242e646c: Generated new TSK 0
I20260925 04:44:34.582715 4549 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/krb5kdc/kudu_127.0.98.65.keytab.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/krb5kdc/kudu_127.0.98.65.keytab.
I20260925 04:44:34.618203 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:0
--local_ip_for_outbound_sockets=127.0.98.65
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:41901
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:39589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:44:34.751137 4560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:44:34.751444 4560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:44:34.751498 4560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:44:34.755880 4560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:44:34.755955 4560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:44:34.756058 4560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260925 04:44:34.761719 4560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:39589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:41901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4560
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:44:34.762934 4560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:44:34.764016 4560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:44:34.774396 4568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:34.774405 4565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:34.774619 4560 server_base.cc:1034] running on GCE node
W20260925 04:44:34.774397 4566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:34.775385 4560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:44:34.776373 4560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:44:34.777614 4560 hybrid_clock.cc:648] HybridClock initialized: now 1790311474777570 us; error 59 us; skew 500 ppm
Sep 25 04:44:34 dist-test-slave-kghz krb5kdc[4454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311474, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:44:34.781766 4560 init.cc:377] Logged in from keytab as kudu/127.0.98.65@KRBTEST.COM (short username kudu)
I20260925 04:44:34.782742 4560 fs_manager.cc:359] Metadata directory not provided
I20260925 04:44:34.782850 4560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:44:34.783119 4560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:44:34.785365 4560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/minicluster-data/ts-0/data/instance:
uuid: "0ea9cb5550fc4dffa85b15aafa8d4b2c"
format_stamp: "Formatted at 2026-09-25 04:44:34 on dist-test-slave-kghz"
server_key: "252aee56aa954b2485a1e67d64fe645a"
server_key_iv: "c7ed06868566c23451bf562f4ae325ec"
server_key_version: "encryptionkey@0"
I20260925 04:44:34.786051 4560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/minicluster-data/ts-0/wal/instance:
uuid: "0ea9cb5550fc4dffa85b15aafa8d4b2c"
format_stamp: "Formatted at 2026-09-25 04:44:34 on dist-test-slave-kghz"
server_key: "252aee56aa954b2485a1e67d64fe645a"
server_key_iv: "c7ed06868566c23451bf562f4ae325ec"
server_key_version: "encryptionkey@0"
I20260925 04:44:34.790364 4560 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 04:44:34.793308 4574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:34.794869 4560 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:44:34.795027 4560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/minicluster-data/ts-0/wal
uuid: "0ea9cb5550fc4dffa85b15aafa8d4b2c"
format_stamp: "Formatted at 2026-09-25 04:44:34 on dist-test-slave-kghz"
server_key: "252aee56aa954b2485a1e67d64fe645a"
server_key_iv: "c7ed06868566c23451bf562f4ae325ec"
server_key_version: "encryptionkey@0"
I20260925 04:44:34.795157 4560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:44:34.811993 4560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:44:34.815770 4560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:44:34.816000 4560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:44:34.816774 4560 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:44:34.817863 4560 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:44:34.817967 4560 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:34.818044 4560 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:44:34.818095 4560 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:34.829746 4560 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:34543
I20260925 04:44:34.829762 4687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:34543 every 8 connection(s)
I20260925 04:44:34.830615 4560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
I20260925 04:44:34.836910 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 4560
I20260925 04:44:34.837060 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/minicluster-data/ts-0/wal/instance
I20260925 04:44:34.837419 393 external_mini_cluster.cc:1428] Setting key 0f00c47c80bf610eaf8bcc574ed44e70
Sep 25 04:44:34 dist-test-slave-kghz krb5kdc[4454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311474, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:44:34.847332 4537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:34.833053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:50241 (local address 127.0.98.126:41901)
0925 04:44:34.833227 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:34.833232 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:34.834264 (+ 1032us) server_negotiation.cc:407] Connection header received
0925 04:44:34.835700 (+ 1436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:34.835706 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:34.835778 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:34.835898 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:34.837251 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:34.837922 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:34.840031 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:34.840470 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:34.840617 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:34.843424 (+ 2807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:34.843428 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:34.843430 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:34.843455 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:34.844972 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:34.845549 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:34.845557 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:34.845558 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:34.845634 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:34.846019 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:34.846022 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:34.846024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:34.846220 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:34.846396 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:34.847011 (+ 615us) server_negotiation.cc:299] Negotiation successful
0925 04:44:34.847163 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 04:44:34.848450 4690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:34.833488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:41901 (local address 127.0.98.65:50241)
0925 04:44:34.834050 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:34.834107 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:34.835477 (+ 1370us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:34.836091 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:34.836099 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:34.836467 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:34.837025 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:34.837049 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:34.838087 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:34.838092 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:34.839858 (+ 1766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:34.839876 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:34.840664 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:34.840668 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:34.840862 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:34.841524 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:34.841542 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:34.843140 (+ 1598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:44:34.845112 (+ 1972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:34.845119 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:34.845138 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:34.845431 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:34.845741 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:34.845744 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:34.845746 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:34.845911 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:34.846400 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:44:34.846411 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:44:34.846741 (+ 330us) client_negotiation.cc:769] Sending connection context
0925 04:44:34.846958 (+ 217us) client_negotiation.cc:240] Negotiation successful
0925 04:44:34.847332 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":145,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.66@KRBTEST.COM; defaulting to no policy
I20260925 04:44:34.849985 4688 heartbeater.cc:347] Connected to a master server at 127.0.98.126:41901
I20260925 04:44:34.850313 4688 heartbeater.cc:464] Registering TS with master...
I20260925 04:44:34.850947 4688 heartbeater.cc:511] Master 127.0.98.126:41901 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.66@KRBTEST.COM" created.
I20260925 04:44:34.854178 4498 ts_manager.cc:194] Registered new tserver with Master: 0ea9cb5550fc4dffa85b15aafa8d4b2c (127.0.98.65:34543)
I20260925 04:44:34.855583 4498 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.65@KRBTEST.COM'} at 127.0.98.65:50241
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/krb5kdc/kudu_127.0.98.66.keytab.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/krb5kdc/kudu_127.0.98.66.keytab.
I20260925 04:44:34.873539 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:0
--local_ip_for_outbound_sockets=127.0.98.66
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:41901
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:39589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:44:35.017643 4693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:44:35.017920 4693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:44:35.017968 4693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:44:35.022182 4693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:44:35.022315 4693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:44:35.022390 4693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260925 04:44:35.027803 4693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:39589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:41901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:44:35.028889 4693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:44:35.029927 4693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:44:35.040041 4699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:35.040227 4701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:35.040087 4698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:35.040055 4693 server_base.cc:1034] running on GCE node
I20260925 04:44:35.040789 4693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:44:35.041533 4693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:44:35.042699 4693 hybrid_clock.cc:648] HybridClock initialized: now 1790311475042680 us; error 32 us; skew 500 ppm
Sep 25 04:44:35 dist-test-slave-kghz krb5kdc[4454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311475, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:44:35.045614 4693 init.cc:377] Logged in from keytab as kudu/127.0.98.66@KRBTEST.COM (short username kudu)
I20260925 04:44:35.046490 4693 fs_manager.cc:359] Metadata directory not provided
I20260925 04:44:35.046561 4693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:44:35.046782 4693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:44:35.049278 4693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/minicluster-data/ts-1/data/instance:
uuid: "f2a7aef2523445e7a040f6d5833caecf"
format_stamp: "Formatted at 2026-09-25 04:44:35 on dist-test-slave-kghz"
server_key: "cf63c20168f577551846e69c5b9a4185"
server_key_iv: "a88d7853b87328dd9984129b7e8c0cd9"
server_key_version: "encryptionkey@0"
I20260925 04:44:35.049851 4693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/minicluster-data/ts-1/wal/instance:
uuid: "f2a7aef2523445e7a040f6d5833caecf"
format_stamp: "Formatted at 2026-09-25 04:44:35 on dist-test-slave-kghz"
server_key: "cf63c20168f577551846e69c5b9a4185"
server_key_iv: "a88d7853b87328dd9984129b7e8c0cd9"
server_key_version: "encryptionkey@0"
I20260925 04:44:35.053736 4693 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 04:44:35.056324 4707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:35.057832 4693 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 04:44:35.057974 4693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/minicluster-data/ts-1/wal
uuid: "f2a7aef2523445e7a040f6d5833caecf"
format_stamp: "Formatted at 2026-09-25 04:44:35 on dist-test-slave-kghz"
server_key: "cf63c20168f577551846e69c5b9a4185"
server_key_iv: "a88d7853b87328dd9984129b7e8c0cd9"
server_key_version: "encryptionkey@0"
I20260925 04:44:35.058105 4693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:44:35.070724 4693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:44:35.074057 4693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:44:35.074321 4693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:44:35.074950 4693 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:44:35.075814 4693 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:44:35.075860 4693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:35.075937 4693 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:44:35.075975 4693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:35.085395 4693 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:34743
I20260925 04:44:35.085413 4820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:34743 every 8 connection(s)
I20260925 04:44:35.086009 4693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
I20260925 04:44:35.091789 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 4693
I20260925 04:44:35.091897 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/minicluster-data/ts-1/wal/instance
I20260925 04:44:35.092208 393 external_mini_cluster.cc:1428] Setting key e549e82b42df5d7f326cccb671b06baf
Sep 25 04:44:35 dist-test-slave-kghz krb5kdc[4454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311475, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:44:35.101480 4537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:35.087821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:45183 (local address 127.0.98.126:41901)
0925 04:44:35.087971 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:35.087974 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:35.088787 (+ 813us) server_negotiation.cc:407] Connection header received
0925 04:44:35.090193 (+ 1406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:35.090197 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:35.090284 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:35.090388 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:35.091598 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:35.092372 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:35.093894 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:35.094138 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:35.094248 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:35.097060 (+ 2812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:35.097063 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:35.097065 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:35.097090 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:35.099094 (+ 2004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:35.099674 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:35.099680 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:35.099685 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:35.099796 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:35.100236 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:35.100242 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:35.100246 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:35.100506 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:35.100628 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:35.101147 (+ 519us) server_negotiation.cc:299] Negotiation successful
0925 04:44:35.101295 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
WARNING: no policy specified for kudu/127.0.98.67@KRBTEST.COM; defaulting to no policy
I20260925 04:44:35.102366 4823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:35.088201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:41901 (local address 127.0.98.66:45183)
0925 04:44:35.088641 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:35.088681 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:35.089977 (+ 1296us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:35.090571 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:35.090578 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:35.090939 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:35.091456 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:35.091471 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:35.092507 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:35.092511 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:35.093777 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:35.093786 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:35.094313 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:35.094318 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:35.094526 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:35.095220 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:35.095236 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:35.096839 (+ 1603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:44:35.099255 (+ 2416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:35.099261 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:35.099276 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:35.099557 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:35.099981 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:35.099983 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:35.099985 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:35.100134 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:35.100620 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:44:35.100625 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:44:35.100892 (+ 267us) client_negotiation.cc:769] Sending connection context
0925 04:44:35.101057 (+ 165us) client_negotiation.cc:240] Negotiation successful
0925 04:44:35.101359 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":113,"threads_started":1}
I20260925 04:44:35.103710 4821 heartbeater.cc:347] Connected to a master server at 127.0.98.126:41901
I20260925 04:44:35.103989 4821 heartbeater.cc:464] Registering TS with master...
I20260925 04:44:35.104561 4821 heartbeater.cc:511] Master 127.0.98.126:41901 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.67@KRBTEST.COM" created.
I20260925 04:44:35.105788 4498 ts_manager.cc:194] Registered new tserver with Master: f2a7aef2523445e7a040f6d5833caecf (127.0.98.66:34743)
I20260925 04:44:35.106720 4498 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.66@KRBTEST.COM'} at 127.0.98.66:45183
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/krb5kdc/kudu_127.0.98.67.keytab.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/krb5kdc/kudu_127.0.98.67.keytab.
I20260925 04:44:35.125924 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:0
--local_ip_for_outbound_sockets=127.0.98.67
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:41901
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:39589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:44:35.242504 4826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:44:35.242841 4826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:44:35.242923 4826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:44:35.247239 4826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:44:35.247344 4826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:44:35.247462 4826 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260925 04:44:35.253257 4826 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:39589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:41901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:44:35.254545 4826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:44:35.255712 4826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:44:35.265714 4834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:35.265714 4831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:35.265790 4826 server_base.cc:1034] running on GCE node
W20260925 04:44:35.265714 4832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:35.266587 4826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:44:35.267259 4826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:44:35.268430 4826 hybrid_clock.cc:648] HybridClock initialized: now 1790311475268413 us; error 31 us; skew 500 ppm
Sep 25 04:44:35 dist-test-slave-kghz krb5kdc[4454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311475, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:44:35.271431 4826 init.cc:377] Logged in from keytab as kudu/127.0.98.67@KRBTEST.COM (short username kudu)
I20260925 04:44:35.272298 4826 fs_manager.cc:359] Metadata directory not provided
I20260925 04:44:35.272363 4826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:44:35.272581 4826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:44:35.274785 4826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/minicluster-data/ts-2/data/instance:
uuid: "f920ea9fd8604ddda26190a960bbc335"
format_stamp: "Formatted at 2026-09-25 04:44:35 on dist-test-slave-kghz"
server_key: "85e11b6e7cf05c6b9186d11bea844000"
server_key_iv: "2367e3a698b5cd6dd36a567f4d2da8c9"
server_key_version: "encryptionkey@0"
I20260925 04:44:35.275384 4826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/minicluster-data/ts-2/wal/instance:
uuid: "f920ea9fd8604ddda26190a960bbc335"
format_stamp: "Formatted at 2026-09-25 04:44:35 on dist-test-slave-kghz"
server_key: "85e11b6e7cf05c6b9186d11bea844000"
server_key_iv: "2367e3a698b5cd6dd36a567f4d2da8c9"
server_key_version: "encryptionkey@0"
I20260925 04:44:35.279335 4826 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 04:44:35.281980 4840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:35.283444 4826 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:44:35.283618 4826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/minicluster-data/ts-2/wal
uuid: "f920ea9fd8604ddda26190a960bbc335"
format_stamp: "Formatted at 2026-09-25 04:44:35 on dist-test-slave-kghz"
server_key: "85e11b6e7cf05c6b9186d11bea844000"
server_key_iv: "2367e3a698b5cd6dd36a567f4d2da8c9"
server_key_version: "encryptionkey@0"
I20260925 04:44:35.283751 4826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:44:35.308998 4826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:44:35.312579 4826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:44:35.312804 4826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:44:35.313444 4826 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:44:35.314396 4826 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:44:35.314474 4826 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:35.314550 4826 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:44:35.314601 4826 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:35.323956 4826 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:33047
I20260925 04:44:35.323971 4953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:33047 every 8 connection(s)
I20260925 04:44:35.324623 4826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
I20260925 04:44:35.333330 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 4826
I20260925 04:44:35.333489 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/minicluster-data/ts-2/wal/instance
I20260925 04:44:35.333838 393 external_mini_cluster.cc:1428] Setting key afcb314456da7641bbacfb31c0ae6a2a
Sep 25 04:44:35 dist-test-slave-kghz krb5kdc[4454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311475, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:44:35.339447 4537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:35.326460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:38123 (local address 127.0.98.126:41901)
0925 04:44:35.326621 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:35.326625 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:35.327584 (+ 959us) server_negotiation.cc:407] Connection header received
0925 04:44:35.329096 (+ 1512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:35.329100 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:35.329167 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:35.329287 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:35.330605 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:35.331141 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:35.332348 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:35.332587 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:35.332708 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:35.335912 (+ 3204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:35.335916 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:35.335919 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:35.335949 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:35.337400 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:35.337944 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:35.337947 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:35.337948 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:35.338023 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:35.338369 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:35.338371 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:35.338372 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:35.338560 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:35.338661 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:35.339209 (+ 548us) server_negotiation.cc:299] Negotiation successful
0925 04:44:35.339324 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 04:44:35.340402 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:35.326893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:41901 (local address 127.0.98.67:38123)
0925 04:44:35.327399 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:35.327449 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:35.328841 (+ 1392us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:35.329477 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:35.329486 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:35.329877 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:35.330435 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:35.330450 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:35.331277 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:35.331283 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:35.332203 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:35.332212 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:35.332757 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:35.332762 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:35.332949 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:35.333726 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:35.333749 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:35.335741 (+ 1992us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:44:35.337537 (+ 1796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:35.337543 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:35.337558 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:35.337840 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:35.338114 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:35.338117 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:35.338119 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:35.338293 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:35.338681 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:44:35.338687 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:44:35.338966 (+ 279us) client_negotiation.cc:769] Sending connection context
0925 04:44:35.339145 (+ 179us) client_negotiation.cc:240] Negotiation successful
0925 04:44:35.339431 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":121,"threads_started":1}
I20260925 04:44:35.341786 4954 heartbeater.cc:347] Connected to a master server at 127.0.98.126:41901
I20260925 04:44:35.342103 4954 heartbeater.cc:464] Registering TS with master...
I20260925 04:44:35.342622 4954 heartbeater.cc:511] Master 127.0.98.126:41901 requested a full tablet report, sending...
I20260925 04:44:35.343670 4498 ts_manager.cc:194] Registered new tserver with Master: f920ea9fd8604ddda26190a960bbc335 (127.0.98.67:33047)
I20260925 04:44:35.344300 4498 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.67@KRBTEST.COM'} at 127.0.98.67:38123
I20260925 04:44:35.348464 393 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 04:44:35.358547 4537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:35.350621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35476 (local address 127.0.98.126:41901)
0925 04:44:35.350799 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:35.350804 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:35.350951 (+ 147us) server_negotiation.cc:407] Connection header received
0925 04:44:35.351124 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:35.351128 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:35.351181 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:35.351274 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:35.352001 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:35.352548 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:35.353409 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:35.353642 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:35.353756 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:35.355218 (+ 1462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:35.355222 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:35.355225 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:35.355256 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:35.356750 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:35.357230 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:35.357235 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:35.357237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:35.357322 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:35.357594 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:35.357597 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:35.357599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:35.357836 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:35.357981 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:35.358218 (+ 237us) server_negotiation.cc:299] Negotiation successful
0925 04:44:35.358367 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 04:44:35.361817 4498 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35476:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:44:35.365038 4498 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:44:35.380705 4968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:35.374527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:34743 (local address 127.0.0.1:37400)
0925 04:44:35.375544 (+ 1017us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:35.375602 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:35.375748 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:35.376382 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:35.376385 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:35.376403 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:44:35.376553 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:35.376562 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:35.377705 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:35.377711 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:35.379036 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:35.379045 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:35.380094 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:35.380098 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:35.380208 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:35.380252 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 04:44:35.380376 (+ 124us) client_negotiation.cc:240] Negotiation successful
0925 04:44:35.380530 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":874,"mutex_wait_us":100,"spinlock_wait_cycles":40064,"thread_start_us":103,"threads_started":1}
I20260925 04:44:35.380717 4969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:35.375028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37400 (local address 127.0.98.66:34743)
0925 04:44:35.376013 (+ 985us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:35.376022 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:35.376041 (+ 19us) server_negotiation.cc:407] Connection header received
0925 04:44:35.376098 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:35.376101 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:35.376240 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:35.376392 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:44:35.376681 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:35.377582 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:35.379167 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:35.379952 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:35.380072 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:35.380233 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:35.380427 (+ 194us) server_negotiation.cc:299] Negotiation successful
0925 04:44:35.380549 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":893,"thread_start_us":142,"threads_started":1}
I20260925 04:44:35.381078 4966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:35.374314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:34543 (local address 127.0.0.1:43040)
0925 04:44:35.375544 (+ 1230us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:35.375602 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:35.375748 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:35.376265 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:35.376271 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:35.376292 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:44:35.376428 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:35.376435 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:35.378093 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:35.378096 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:35.379359 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:35.379369 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:35.380647 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:35.380651 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:35.380749 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:35.380794 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 04:44:35.380873 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 04:44:35.380947 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1040,"spinlock_wait_cycles":20992,"thread_start_us":81,"threads_started":1}
I20260925 04:44:35.381495 4965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:35.374702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:33047 (local address 127.0.0.1:43698)
0925 04:44:35.375544 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:35.375602 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:35.375748 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:35.377016 (+ 1268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:35.377019 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:35.377037 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:44:35.377123 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:35.377128 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:35.378439 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:35.378442 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:35.379565 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:35.379572 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:35.380711 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:35.380717 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:35.381040 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:35.381063 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 04:44:35.381139 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 04:44:35.381399 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":750,"mutex_wait_us":1,"spinlock_wait_cycles":27264,"thread_start_us":104,"threads_started":1}
I20260925 04:44:35.381551 4967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:35.374519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43040 (local address 127.0.98.65:34543)
0925 04:44:35.375854 (+ 1335us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:35.375861 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:35.375882 (+ 21us) server_negotiation.cc:407] Connection header received
0925 04:44:35.375942 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:35.375946 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:35.376115 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:35.376266 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:44:35.376562 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:35.377945 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:35.379495 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:35.380317 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:35.380447 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:35.380579 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:35.381282 (+ 703us) server_negotiation.cc:299] Negotiation successful
0925 04:44:35.381390 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":680,"thread_start_us":128,"threads_started":1}
I20260925 04:44:35.382460 4970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:35.375139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43698 (local address 127.0.98.67:33047)
0925 04:44:35.376567 (+ 1428us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:35.376580 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:35.376600 (+ 20us) server_negotiation.cc:407] Connection header received
0925 04:44:35.376672 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:35.376677 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:35.376841 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:35.377010 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:44:35.377220 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:35.378300 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:35.379695 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:35.380426 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:35.380525 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:35.380652 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:35.382129 (+ 1477us) server_negotiation.cc:299] Negotiation successful
0925 04:44:35.382266 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1320,"thread_start_us":125,"threads_started":1}
I20260925 04:44:35.384443 4755 tablet_service.cc:1467] Processing CreateTablet for tablet 3fb4dcff7e744ba8bad60c7e5cda8f34 (DEFAULT_TABLE table=test-table [id=c168725237ad496aafe49e9bd34354a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:44:35.385170 4888 tablet_service.cc:1467] Processing CreateTablet for tablet 3fb4dcff7e744ba8bad60c7e5cda8f34 (DEFAULT_TABLE table=test-table [id=c168725237ad496aafe49e9bd34354a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:44:35.385658 4755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fb4dcff7e744ba8bad60c7e5cda8f34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:44:35.386287 4888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fb4dcff7e744ba8bad60c7e5cda8f34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:44:35.386945 4622 tablet_service.cc:1467] Processing CreateTablet for tablet 3fb4dcff7e744ba8bad60c7e5cda8f34 (DEFAULT_TABLE table=test-table [id=c168725237ad496aafe49e9bd34354a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:44:35.387970 4622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fb4dcff7e744ba8bad60c7e5cda8f34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:44:35.393360 4972 tablet_bootstrap.cc:492] T 3fb4dcff7e744ba8bad60c7e5cda8f34 P f2a7aef2523445e7a040f6d5833caecf: Bootstrap starting.
I20260925 04:44:35.393810 4971 tablet_bootstrap.cc:492] T 3fb4dcff7e744ba8bad60c7e5cda8f34 P f920ea9fd8604ddda26190a960bbc335: Bootstrap starting.
I20260925 04:44:35.394604 4973 tablet_bootstrap.cc:492] T 3fb4dcff7e744ba8bad60c7e5cda8f34 P 0ea9cb5550fc4dffa85b15aafa8d4b2c: Bootstrap starting.
I20260925 04:44:35.396301 4971 tablet_bootstrap.cc:654] T 3fb4dcff7e744ba8bad60c7e5cda8f34 P f920ea9fd8604ddda26190a960bbc335: Neither blocks nor log segments found. Creating new log.
I20260925 04:44:35.396394 4973 tablet_bootstrap.cc:654] T 3fb4dcff7e744ba8bad60c7e5cda8f34 P 0ea9cb5550fc4dffa85b15aafa8d4b2c: Neither blocks nor log segments found. Creating new log.
I20260925 04:44:35.397219 4972 tablet_bootstrap.cc:654] T 3fb4dcff7e744ba8bad60c7e5cda8f34 P f2a7aef2523445e7a040f6d5833caecf: Neither blocks nor log segments found. Creating new log.
I20260925 04:44:35.397280 4973 log.cc:824] T 3fb4dcff7e744ba8bad60c7e5cda8f34 P 0ea9cb5550fc4dffa85b15aafa8d4b2c: Log is configured to *not* fsync() on all Append() calls
I20260925 04:44:35.397643 4971 log.cc:824] T 3fb4dcff7e744ba8bad60c7e5cda8f34 P f920ea9fd8604ddda26190a960bbc335: Log is configured to *not* fsync() on all Append() calls
I20260925 04:44:35.398113 4972 log.cc:824] T 3fb4dcff7e744ba8bad60c7e5cda8f34 P f2a7aef2523445e7a040f6d5833caecf: Log is configured to *not* fsync() on all Append() calls
I20260925 04:44:35.399629 4971 tablet_bootstrap.cc:492] T 3fb4dcff7e744ba8bad60c7e5cda8f34 P f920ea9fd8604ddda26190a960bbc335: No bootstrap required, opened a new log
I20260925 04:44:35.399641 4973 tablet_bootstrap.cc:492] T 3fb4dcff7e744ba8bad60c7e5cda8f34 P 0ea9cb5550fc4dffa85b15aafa8d4b2c: No bootstrap required, opened a new log
I20260925 04:44:35.399853 4971 ts_tablet_manager.cc:1397] T 3fb4dcff7e744ba8bad60c7e5cda8f34 P f920ea9fd8604ddda26190a960bbc335: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 04:44:35.399855 4973 ts_tablet_manager.cc:1397] T 3fb4dcff7e744ba8bad60c7e5cda8f34 P 0ea9cb5550fc4dffa85b15aafa8d4b2c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 04:44:35.400242 4972 tablet_bootstrap.cc:492] T 3fb4dcff7e744ba8bad60c7e5cda8f34 P f2a7aef2523445e7a040f6d5833caecf: No bootstrap required, opened a new log
I20260925 04:44:35.400422 4972 ts_tablet_manager.cc:1397] T 3fb4dcff7e744ba8bad60c7e5cda8f34 P f2a7aef2523445e7a040f6d5833caecf: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 04:44:35.403033 4971 raft_consensus.cc:348] T 3fb4dcff7e744ba8bad60c7e5cda8f34 P f920ea9fd8604ddda26190a960bbc335 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea9cb5550fc4dffa85b15aafa8d4b2c" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34543 } } peers { permanent_uuid: "f2a7aef2523445e7a040f6d5833caecf" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 34743 } } peers { permanent_uuid: "f920ea9fd8604ddda26190a960bbc335" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33047 } }
I20260925 04:44:35.403307 4971 raft_consensus.cc:374] T 3fb4dcff7e744ba8bad60c7e5cda8f34 P f920ea9fd8604ddda26190a960bbc335 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:44:35.403383 4971 raft_consensus.cc:729] T 3fb4dcff7e744ba8bad60c7e5cda8f34 P f920ea9fd8604ddda26190a960bbc335 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f920ea9fd8604ddda26190a960bbc335, State: Initialized, Role: FOLLOWER
I20260925 04:44:35.403899 4971 consensus_queue.cc:260] T 3fb4dcff7e744ba8bad60c7e5cda8f34 P f920ea9fd8604ddda26190a960bbc335 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea9cb5550fc4dffa85b15aafa8d4b2c" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34543 } } peers { permanent_uuid: "f2a7aef2523445e7a040f6d5833caecf" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 34743 } } peers { permanent_uuid: "f920ea9fd8604ddda26190a960bbc335" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33047 } }
I20260925 04:44:35.404421 4973 raft_consensus.cc:348] T 3fb4dcff7e744ba8bad60c7e5cda8f34 P 0ea9cb5550fc4dffa85b15aafa8d4b2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea9cb5550fc4dffa85b15aafa8d4b2c" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34543 } } peers { permanent_uuid: "f2a7aef2523445e7a040f6d5833caecf" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 34743 } } peers { permanent_uuid: "f920ea9fd8604ddda26190a960bbc335" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33047 } }
I20260925 04:44:35.404421 4972 raft_consensus.cc:348] T 3fb4dcff7e744ba8bad60c7e5cda8f34 P f2a7aef2523445e7a040f6d5833caecf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea9cb5550fc4dffa85b15aafa8d4b2c" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34543 } } peers { permanent_uuid: "f2a7aef2523445e7a040f6d5833caecf" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 34743 } } peers { permanent_uuid: "f920ea9fd8604ddda26190a960bbc335" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33047 } }
I20260925 04:44:35.404686 4972 raft_consensus.cc:374] T 3fb4dcff7e744ba8bad60c7e5cda8f34 P f2a7aef2523445e7a040f6d5833caecf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:44:35.404731 4973 raft_consensus.cc:374] T 3fb4dcff7e744ba8bad60c7e5cda8f34 P 0ea9cb5550fc4dffa85b15aafa8d4b2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:44:35.404778 4972 raft_consensus.cc:729] T 3fb4dcff7e744ba8bad60c7e5cda8f34 P f2a7aef2523445e7a040f6d5833caecf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2a7aef2523445e7a040f6d5833caecf, State: Initialized, Role: FOLLOWER
I20260925 04:44:35.404812 4973 raft_consensus.cc:729] T 3fb4dcff7e744ba8bad60c7e5cda8f34 P 0ea9cb5550fc4dffa85b15aafa8d4b2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ea9cb5550fc4dffa85b15aafa8d4b2c, State: Initialized, Role: FOLLOWER
I20260925 04:44:35.405251 4971 ts_tablet_manager.cc:1428] T 3fb4dcff7e744ba8bad60c7e5cda8f34 P f920ea9fd8604ddda26190a960bbc335: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260925 04:44:35.405258 4973 consensus_queue.cc:260] T 3fb4dcff7e744ba8bad60c7e5cda8f34 P 0ea9cb5550fc4dffa85b15aafa8d4b2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea9cb5550fc4dffa85b15aafa8d4b2c" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34543 } } peers { permanent_uuid: "f2a7aef2523445e7a040f6d5833caecf" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 34743 } } peers { permanent_uuid: "f920ea9fd8604ddda26190a960bbc335" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33047 } }
I20260925 04:44:35.405258 4972 consensus_queue.cc:260] T 3fb4dcff7e744ba8bad60c7e5cda8f34 P f2a7aef2523445e7a040f6d5833caecf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea9cb5550fc4dffa85b15aafa8d4b2c" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34543 } } peers { permanent_uuid: "f2a7aef2523445e7a040f6d5833caecf" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 34743 } } peers { permanent_uuid: "f920ea9fd8604ddda26190a960bbc335" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33047 } }
I20260925 04:44:35.405661 4954 heartbeater.cc:503] Master 127.0.98.126:41901 was elected leader, sending a full tablet report...
I20260925 04:44:35.406229 4973 ts_tablet_manager.cc:1428] T 3fb4dcff7e744ba8bad60c7e5cda8f34 P 0ea9cb5550fc4dffa85b15aafa8d4b2c: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 04:44:35.406229 4972 ts_tablet_manager.cc:1428] T 3fb4dcff7e744ba8bad60c7e5cda8f34 P f2a7aef2523445e7a040f6d5833caecf: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260925 04:44:35.406324 4688 heartbeater.cc:503] Master 127.0.98.126:41901 was elected leader, sending a full tablet report...
I20260925 04:44:35.407089 4821 heartbeater.cc:503] Master 127.0.98.126:41901 was elected leader, sending a full tablet report...
I20260925 04:44:35.423676 4978 raft_consensus.cc:482] T 3fb4dcff7e744ba8bad60c7e5cda8f34 P 0ea9cb5550fc4dffa85b15aafa8d4b2c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:44:35.423926 4978 raft_consensus.cc:504] T 3fb4dcff7e744ba8bad60c7e5cda8f34 P 0ea9cb5550fc4dffa85b15aafa8d4b2c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea9cb5550fc4dffa85b15aafa8d4b2c" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34543 } } peers { permanent_uuid: "f2a7aef2523445e7a040f6d5833caecf" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 34743 } } peers { permanent_uuid: "f920ea9fd8604ddda26190a960bbc335" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33047 } }
I20260925 04:44:35.425158 4978 leader_election.cc:288] T 3fb4dcff7e744ba8bad60c7e5cda8f34 P 0ea9cb5550fc4dffa85b15aafa8d4b2c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f2a7aef2523445e7a040f6d5833caecf (127.0.98.66:34743), f920ea9fd8604ddda26190a960bbc335 (127.0.98.67:33047)
I20260925 04:44:35.429126 4970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:35.425544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:44011 (local address 127.0.98.67:33047)
0925 04:44:35.425713 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:35.425719 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:35.425853 (+ 134us) server_negotiation.cc:407] Connection header received
0925 04:44:35.425988 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:35.425993 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:35.426042 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:35.426134 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:44:35.426441 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:35.426949 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:35.428108 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:35.428517 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:35.428656 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:35.428721 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:35.428903 (+ 182us) server_negotiation.cc:299] Negotiation successful
0925 04:44:35.428981 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260925 04:44:35.429162 4969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:35.425435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:47953 (local address 127.0.98.66:34743)
0925 04:44:35.425587 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:35.425592 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:35.425912 (+ 320us) server_negotiation.cc:407] Connection header received
0925 04:44:35.426102 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:35.426104 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:35.426147 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:35.426239 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:44:35.426517 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:35.427017 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:35.428106 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:35.428513 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:35.428675 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:35.428740 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:35.428958 (+ 218us) server_negotiation.cc:299] Negotiation successful
0925 04:44:35.429040 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 04:44:35.429179 4981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:35.425352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:33047 (local address 127.0.98.65:44011)
0925 04:44:35.425754 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:35.425767 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:35.425866 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:35.426144 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:35.426147 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:35.426162 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:44:35.426329 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:35.426335 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:35.427064 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:35.427069 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:35.427986 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:35.427995 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:35.428665 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:35.428668 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:35.428761 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:35.428791 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 04:44:35.428875 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 04:44:35.428953 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"mutex_wait_us":48,"thread_start_us":168,"threads_started":1}
I20260925 04:44:35.429188 4980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:35.425357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:34743 (local address 127.0.98.65:47953)
0925 04:44:35.425808 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:35.425841 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:35.425952 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:35.426267 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:35.426270 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:35.426287 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:44:35.426406 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:35.426412 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:35.427134 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:35.427139 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:35.427974 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:35.427982 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:35.428687 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:35.428691 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:35.428816 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:35.428846 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 04:44:35.428931 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 04:44:35.429005 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"mutex_wait_us":36,"thread_start_us":142,"threads_started":1}
I20260925 04:44:35.429965 4908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fb4dcff7e744ba8bad60c7e5cda8f34" candidate_uuid: "0ea9cb5550fc4dffa85b15aafa8d4b2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f920ea9fd8604ddda26190a960bbc335" is_pre_election: true
I20260925 04:44:35.430100 4775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fb4dcff7e744ba8bad60c7e5cda8f34" candidate_uuid: "0ea9cb5550fc4dffa85b15aafa8d4b2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2a7aef2523445e7a040f6d5833caecf" is_pre_election: true
I20260925 04:44:35.430393 4908 raft_consensus.cc:2454] T 3fb4dcff7e744ba8bad60c7e5cda8f34 P f920ea9fd8604ddda26190a960bbc335 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ea9cb5550fc4dffa85b15aafa8d4b2c in term 0.
I20260925 04:44:35.430438 4775 raft_consensus.cc:2454] T 3fb4dcff7e744ba8bad60c7e5cda8f34 P f2a7aef2523445e7a040f6d5833caecf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ea9cb5550fc4dffa85b15aafa8d4b2c in term 0.
I20260925 04:44:35.430940 4578 leader_election.cc:302] T 3fb4dcff7e744ba8bad60c7e5cda8f34 P 0ea9cb5550fc4dffa85b15aafa8d4b2c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ea9cb5550fc4dffa85b15aafa8d4b2c, f920ea9fd8604ddda26190a960bbc335; no voters:
I20260925 04:44:35.431231 4978 raft_consensus.cc:2781] T 3fb4dcff7e744ba8bad60c7e5cda8f34 P 0ea9cb5550fc4dffa85b15aafa8d4b2c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:44:35.431325 4978 raft_consensus.cc:482] T 3fb4dcff7e744ba8bad60c7e5cda8f34 P 0ea9cb5550fc4dffa85b15aafa8d4b2c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:44:35.431383 4978 raft_consensus.cc:3037] T 3fb4dcff7e744ba8bad60c7e5cda8f34 P 0ea9cb5550fc4dffa85b15aafa8d4b2c [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:44:35.432440 4978 raft_consensus.cc:504] T 3fb4dcff7e744ba8bad60c7e5cda8f34 P 0ea9cb5550fc4dffa85b15aafa8d4b2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea9cb5550fc4dffa85b15aafa8d4b2c" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34543 } } peers { permanent_uuid: "f2a7aef2523445e7a040f6d5833caecf" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 34743 } } peers { permanent_uuid: "f920ea9fd8604ddda26190a960bbc335" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33047 } }
I20260925 04:44:35.432818 4978 leader_election.cc:288] T 3fb4dcff7e744ba8bad60c7e5cda8f34 P 0ea9cb5550fc4dffa85b15aafa8d4b2c [CANDIDATE]: Term 1 election: Requested vote from peers f2a7aef2523445e7a040f6d5833caecf (127.0.98.66:34743), f920ea9fd8604ddda26190a960bbc335 (127.0.98.67:33047)
I20260925 04:44:35.433280 4908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fb4dcff7e744ba8bad60c7e5cda8f34" candidate_uuid: "0ea9cb5550fc4dffa85b15aafa8d4b2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f920ea9fd8604ddda26190a960bbc335"
I20260925 04:44:35.433320 4775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fb4dcff7e744ba8bad60c7e5cda8f34" candidate_uuid: "0ea9cb5550fc4dffa85b15aafa8d4b2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2a7aef2523445e7a040f6d5833caecf"
I20260925 04:44:35.433459 4775 raft_consensus.cc:3037] T 3fb4dcff7e744ba8bad60c7e5cda8f34 P f2a7aef2523445e7a040f6d5833caecf [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:44:35.433406 4908 raft_consensus.cc:3037] T 3fb4dcff7e744ba8bad60c7e5cda8f34 P f920ea9fd8604ddda26190a960bbc335 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:44:35.434653 4908 raft_consensus.cc:2454] T 3fb4dcff7e744ba8bad60c7e5cda8f34 P f920ea9fd8604ddda26190a960bbc335 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ea9cb5550fc4dffa85b15aafa8d4b2c in term 1.
I20260925 04:44:35.434662 4775 raft_consensus.cc:2454] T 3fb4dcff7e744ba8bad60c7e5cda8f34 P f2a7aef2523445e7a040f6d5833caecf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ea9cb5550fc4dffa85b15aafa8d4b2c in term 1.
I20260925 04:44:35.435024 4576 leader_election.cc:302] T 3fb4dcff7e744ba8bad60c7e5cda8f34 P 0ea9cb5550fc4dffa85b15aafa8d4b2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ea9cb5550fc4dffa85b15aafa8d4b2c, f2a7aef2523445e7a040f6d5833caecf; no voters:
I20260925 04:44:35.435254 4978 raft_consensus.cc:2781] T 3fb4dcff7e744ba8bad60c7e5cda8f34 P 0ea9cb5550fc4dffa85b15aafa8d4b2c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:44:35.435487 4978 raft_consensus.cc:686] T 3fb4dcff7e744ba8bad60c7e5cda8f34 P 0ea9cb5550fc4dffa85b15aafa8d4b2c [term 1 LEADER]: Becoming Leader. State: Replica: 0ea9cb5550fc4dffa85b15aafa8d4b2c, State: Running, Role: LEADER
I20260925 04:44:35.435876 4978 consensus_queue.cc:237] T 3fb4dcff7e744ba8bad60c7e5cda8f34 P 0ea9cb5550fc4dffa85b15aafa8d4b2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea9cb5550fc4dffa85b15aafa8d4b2c" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34543 } } peers { permanent_uuid: "f2a7aef2523445e7a040f6d5833caecf" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 34743 } } peers { permanent_uuid: "f920ea9fd8604ddda26190a960bbc335" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33047 } }
I20260925 04:44:35.439524 4497 catalog_manager.cc:5495] T 3fb4dcff7e744ba8bad60c7e5cda8f34 P 0ea9cb5550fc4dffa85b15aafa8d4b2c reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ea9cb5550fc4dffa85b15aafa8d4b2c (127.0.98.65). New cstate: current_term: 1 leader_uuid: "0ea9cb5550fc4dffa85b15aafa8d4b2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea9cb5550fc4dffa85b15aafa8d4b2c" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34543 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f2a7aef2523445e7a040f6d5833caecf" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 34743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f920ea9fd8604ddda26190a960bbc335" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33047 } health_report { overall_health: UNKNOWN } } }
I20260925 04:44:35.461678 4967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:35.457977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43052 (local address 127.0.98.65:34543)
0925 04:44:35.458114 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:35.458118 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:35.458131 (+ 13us) server_negotiation.cc:407] Connection header received
0925 04:44:35.458351 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:35.458354 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:35.458411 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:35.458519 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:44:35.458779 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:35.459293 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:35.460149 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:35.460394 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:35.460479 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:35.460770 (+ 291us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:44:35.461210 (+ 440us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:44:35.461309 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:35.461423 (+ 114us) server_negotiation.cc:299] Negotiation successful
0925 04:44:35.461475 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 04:44:35.467999 4775 raft_consensus.cc:1261] T 3fb4dcff7e744ba8bad60c7e5cda8f34 P f2a7aef2523445e7a040f6d5833caecf [term 1 FOLLOWER]: Refusing update from remote peer 0ea9cb5550fc4dffa85b15aafa8d4b2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:44:35.467999 4908 raft_consensus.cc:1261] T 3fb4dcff7e744ba8bad60c7e5cda8f34 P f920ea9fd8604ddda26190a960bbc335 [term 1 FOLLOWER]: Refusing update from remote peer 0ea9cb5550fc4dffa85b15aafa8d4b2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:44:35.468799 4982 consensus_queue.cc:1035] T 3fb4dcff7e744ba8bad60c7e5cda8f34 P 0ea9cb5550fc4dffa85b15aafa8d4b2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2a7aef2523445e7a040f6d5833caecf" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 34743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:44:35.469003 4978 consensus_queue.cc:1035] T 3fb4dcff7e744ba8bad60c7e5cda8f34 P 0ea9cb5550fc4dffa85b15aafa8d4b2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f920ea9fd8604ddda26190a960bbc335" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:44:35.477327 4986 mvcc.cc:205] Tried to move back new op lower bound from 7333115803508936704 to 7333115803388719104. Current Snapshot: MvccSnapshot[applied={T|T < 7333115803508936704}]
I20260925 04:44:35.479113 4987 mvcc.cc:205] Tried to move back new op lower bound from 7333115803508936704 to 7333115803388719104. Current Snapshot: MvccSnapshot[applied={T|T < 7333115803508936704}]
I20260925 04:44:35.486418 4495 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35476:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:44:35.486651 4495 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35476:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:44:35.489713 4495 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 12a0987c0c964debb984f75c242e646c: Sending DeleteTablet for 3 replicas of tablet 3fb4dcff7e744ba8bad60c7e5cda8f34
I20260925 04:44:35.490659 4622 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fb4dcff7e744ba8bad60c7e5cda8f34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:44:35 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:43040
I20260925 04:44:35.490660 4755 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fb4dcff7e744ba8bad60c7e5cda8f34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:44:35 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:37400
I20260925 04:44:35.491019 4888 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fb4dcff7e744ba8bad60c7e5cda8f34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:44:35 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:43698
I20260925 04:44:35.491575 4995 tablet_replica.cc:330] T 3fb4dcff7e744ba8bad60c7e5cda8f34 P 0ea9cb5550fc4dffa85b15aafa8d4b2c: stopping tablet replica
I20260925 04:44:35.491648 4996 tablet_replica.cc:330] T 3fb4dcff7e744ba8bad60c7e5cda8f34 P f2a7aef2523445e7a040f6d5833caecf: stopping tablet replica
I20260925 04:44:35.491966 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 4560
I20260925 04:44:35.492029 4995 raft_consensus.cc:2229] T 3fb4dcff7e744ba8bad60c7e5cda8f34 P 0ea9cb5550fc4dffa85b15aafa8d4b2c [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:44:35.492030 4996 raft_consensus.cc:2229] T 3fb4dcff7e744ba8bad60c7e5cda8f34 P f2a7aef2523445e7a040f6d5833caecf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:44:35.492384 4996 raft_consensus.cc:2258] T 3fb4dcff7e744ba8bad60c7e5cda8f34 P f2a7aef2523445e7a040f6d5833caecf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:44:35.492408 4995 raft_consensus.cc:2258] T 3fb4dcff7e744ba8bad60c7e5cda8f34 P 0ea9cb5550fc4dffa85b15aafa8d4b2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:44:35.492746 4997 tablet_replica.cc:330] T 3fb4dcff7e744ba8bad60c7e5cda8f34 P f920ea9fd8604ddda26190a960bbc335: stopping tablet replica
I20260925 04:44:35.493078 4997 raft_consensus.cc:2229] T 3fb4dcff7e744ba8bad60c7e5cda8f34 P f920ea9fd8604ddda26190a960bbc335 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:44:35.493410 4997 raft_consensus.cc:2258] T 3fb4dcff7e744ba8bad60c7e5cda8f34 P f920ea9fd8604ddda26190a960bbc335 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:44:35.495569 4997 ts_tablet_manager.cc:1905] T 3fb4dcff7e744ba8bad60c7e5cda8f34 P f920ea9fd8604ddda26190a960bbc335: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:44:35.495824 4996 ts_tablet_manager.cc:1905] T 3fb4dcff7e744ba8bad60c7e5cda8f34 P f2a7aef2523445e7a040f6d5833caecf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:44:35.497991 4997 ts_tablet_manager.cc:1918] T 3fb4dcff7e744ba8bad60c7e5cda8f34 P f920ea9fd8604ddda26190a960bbc335: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:44:35.498135 4997 log.cc:1196] T 3fb4dcff7e744ba8bad60c7e5cda8f34 P f920ea9fd8604ddda26190a960bbc335: Deleting WAL directory at /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/minicluster-data/ts-2/wal/wals/3fb4dcff7e744ba8bad60c7e5cda8f34
I20260925 04:44:35.498499 4997 ts_tablet_manager.cc:1939] T 3fb4dcff7e744ba8bad60c7e5cda8f34 P f920ea9fd8604ddda26190a960bbc335: Deleting consensus metadata
I20260925 04:44:35.499466 4996 ts_tablet_manager.cc:1918] T 3fb4dcff7e744ba8bad60c7e5cda8f34 P f2a7aef2523445e7a040f6d5833caecf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:44:35.499583 4996 log.cc:1196] T 3fb4dcff7e744ba8bad60c7e5cda8f34 P f2a7aef2523445e7a040f6d5833caecf: Deleting WAL directory at /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790311449873301-393-0/minicluster-data/ts-1/wal/wals/3fb4dcff7e744ba8bad60c7e5cda8f34
W20260925 04:44:35.499948 4484 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.65:34543 (error 108)
I20260925 04:44:35.500012 4996 ts_tablet_manager.cc:1939] T 3fb4dcff7e744ba8bad60c7e5cda8f34 P f2a7aef2523445e7a040f6d5833caecf: Deleting consensus metadata
I20260925 04:44:35.500301 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 4693
I20260925 04:44:35.500428 4485 catalog_manager.cc:4841] TS f920ea9fd8604ddda26190a960bbc335 (127.0.98.67:33047): tablet 3fb4dcff7e744ba8bad60c7e5cda8f34 (table test-table [id=c168725237ad496aafe49e9bd34354a5]) successfully deleted
I20260925 04:44:35.501417 4965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:35.501068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:34543 (local address 127.0.0.1:43060)
0925 04:44:35.501232 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:35.501312 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.65:34543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 04:44:35.501591 4484 catalog_manager.cc:4584] TS 0ea9cb5550fc4dffa85b15aafa8d4b2c (127.0.98.65:34543): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3fb4dcff7e744ba8bad60c7e5cda8f34: Network error: Client connection negotiation failed: client connection to 127.0.98.65:34543: connect: Connection refused (error 111)
I20260925 04:44:35.506796 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 4826
I20260925 04:44:35.507985 4965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:35.507030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:34743 (local address 127.0.0.1:37410)
0925 04:44:35.507791 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:35.507865 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.66:34743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":636}
I20260925 04:44:35.513245 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 4467
2026-09-25T04:44:35Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2381 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-25T04:44:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:44:35Z Disabled control of system clock
I20260925 04:44:35.545085 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311449873301-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311449873301-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311449873301-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:37817
--webserver_interface=127.0.98.126
--webserver_port=0
--builtin_ntp_servers=127.0.98.84:34513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:37817
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 04:44:35.661530 5002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:44:35.661834 5002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:44:35.661908 5002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:44:35.666435 5002 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:44:35.666517 5002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:44:35.666533 5002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:44:35.666551 5002 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:44:35.666568 5002 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:44:35.671953 5002 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:34513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311449873301-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311449873301-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:37817
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:37817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5002
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311449873301-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:44:35.673169 5002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:44:35.674687 5002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 04:44:35.684690 5002 server_base.cc:1034] running on GCE node
W20260925 04:44:35.684587 5008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:35.684584 5010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:35.684588 5007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:35.686075 5002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:44:35.687400 5002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:44:35.688616 5002 hybrid_clock.cc:648] HybridClock initialized: now 1790311475688604 us; error 36 us; skew 500 ppm
I20260925 04:44:35.690990 5002 webserver.cc:460] Webserver started at http://127.0.98.126:36345/ using document root <none> and password file <none>
I20260925 04:44:35.691620 5002 fs_manager.cc:359] Metadata directory not provided
I20260925 04:44:35.691717 5002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:44:35.692298 5002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:44:35.694880 5002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311449873301-393-0/minicluster-data/master-0/data/instance:
uuid: "66f5bc501a3443f3bce78dc1c6241278"
format_stamp: "Formatted at 2026-09-25 04:44:35 on dist-test-slave-kghz"
server_key: "4e5df9f4aae5ad5305a0cde078ac8e5d"
server_key_iv: "0707ce6037dd88b48bce1dbc28f67213"
server_key_version: "encryptionkey@0"
I20260925 04:44:35.695542 5002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311449873301-393-0/minicluster-data/master-0/wal/instance:
uuid: "66f5bc501a3443f3bce78dc1c6241278"
format_stamp: "Formatted at 2026-09-25 04:44:35 on dist-test-slave-kghz"
server_key: "4e5df9f4aae5ad5305a0cde078ac8e5d"
server_key_iv: "0707ce6037dd88b48bce1dbc28f67213"
server_key_version: "encryptionkey@0"
I20260925 04:44:35.699788 5002 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 04:44:35.702917 5016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:35.704421 5002 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 04:44:35.704608 5002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311449873301-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311449873301-393-0/minicluster-data/master-0/wal
uuid: "66f5bc501a3443f3bce78dc1c6241278"
format_stamp: "Formatted at 2026-09-25 04:44:35 on dist-test-slave-kghz"
server_key: "4e5df9f4aae5ad5305a0cde078ac8e5d"
server_key_iv: "0707ce6037dd88b48bce1dbc28f67213"
server_key_version: "encryptionkey@0"
I20260925 04:44:35.704767 5002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311449873301-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311449873301-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311449873301-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:44:35.709735 5002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:44:35.710489 5002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:44:35.710692 5002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:44:35.718281 5002 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:37817
I20260925 04:44:35.718297 5068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:37817 every 8 connection(s)
I20260925 04:44:35.719324 5002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
I20260925 04:44:35.721509 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 5002
I20260925 04:44:35.721609 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311449873301-393-0/minicluster-data/master-0/wal/instance
I20260925 04:44:35.721916 393 external_mini_cluster.cc:1428] Setting key 6477d3de80cf87792f8ae7ca5286a477
I20260925 04:44:35.722949 5069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:44:35.727946 5072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:35.723574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41190 (local address 127.0.98.126:37817)
0925 04:44:35.724281 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:35.724297 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:35.724334 (+ 37us) server_negotiation.cc:407] Connection header received
0925 04:44:35.725002 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:35.725021 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:35.725380 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:35.725764 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:35.726625 (+ 861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:35.726630 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:35.726643 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:44:35.726654 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:35.726713 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:44:35.726729 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:35.726890 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:35.726966 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 04:44:35.727220 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":160,"threads_started":1}
I20260925 04:44:35.729885 5069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66f5bc501a3443f3bce78dc1c6241278: Bootstrap starting.
I20260925 04:44:35.732394 5069 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66f5bc501a3443f3bce78dc1c6241278: Neither blocks nor log segments found. Creating new log.
I20260925 04:44:35.733292 5069 log.cc:824] T 00000000000000000000000000000000 P 66f5bc501a3443f3bce78dc1c6241278: Log is configured to *not* fsync() on all Append() calls
I20260925 04:44:35.735446 5069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66f5bc501a3443f3bce78dc1c6241278: No bootstrap required, opened a new log
I20260925 04:44:35.739231 5069 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66f5bc501a3443f3bce78dc1c6241278 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66f5bc501a3443f3bce78dc1c6241278" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 37817 } }
I20260925 04:44:35.739498 5069 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66f5bc501a3443f3bce78dc1c6241278 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:44:35.739578 5069 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66f5bc501a3443f3bce78dc1c6241278 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66f5bc501a3443f3bce78dc1c6241278, State: Initialized, Role: FOLLOWER
I20260925 04:44:35.740034 5069 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66f5bc501a3443f3bce78dc1c6241278 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66f5bc501a3443f3bce78dc1c6241278" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 37817 } }
I20260925 04:44:35.740201 5069 raft_consensus.cc:388] T 00000000000000000000000000000000 P 66f5bc501a3443f3bce78dc1c6241278 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:44:35.740293 5069 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66f5bc501a3443f3bce78dc1c6241278 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:44:35.740423 5069 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66f5bc501a3443f3bce78dc1c6241278 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:44:35.741541 5069 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66f5bc501a3443f3bce78dc1c6241278 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66f5bc501a3443f3bce78dc1c6241278" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 37817 } }
I20260925 04:44:35.741868 5069 leader_election.cc:302] T 00000000000000000000000000000000 P 66f5bc501a3443f3bce78dc1c6241278 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66f5bc501a3443f3bce78dc1c6241278; no voters:
I20260925 04:44:35.742200 5069 leader_election.cc:288] T 00000000000000000000000000000000 P 66f5bc501a3443f3bce78dc1c6241278 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:44:35.742373 5074 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66f5bc501a3443f3bce78dc1c6241278 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:44:35.742658 5074 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66f5bc501a3443f3bce78dc1c6241278 [term 1 LEADER]: Becoming Leader. State: Replica: 66f5bc501a3443f3bce78dc1c6241278, State: Running, Role: LEADER
I20260925 04:44:35.743033 5074 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66f5bc501a3443f3bce78dc1c6241278 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66f5bc501a3443f3bce78dc1c6241278" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 37817 } }
I20260925 04:44:35.743605 5069 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66f5bc501a3443f3bce78dc1c6241278 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:44:35.744851 5075 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66f5bc501a3443f3bce78dc1c6241278 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "66f5bc501a3443f3bce78dc1c6241278" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66f5bc501a3443f3bce78dc1c6241278" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 37817 } } }
I20260925 04:44:35.744897 5076 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66f5bc501a3443f3bce78dc1c6241278 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66f5bc501a3443f3bce78dc1c6241278. Latest consensus state: current_term: 1 leader_uuid: "66f5bc501a3443f3bce78dc1c6241278" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66f5bc501a3443f3bce78dc1c6241278" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 37817 } } }
I20260925 04:44:35.744980 5075 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66f5bc501a3443f3bce78dc1c6241278 [sys.catalog]: This master's current role is: LEADER
I20260925 04:44:35.745005 5076 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66f5bc501a3443f3bce78dc1c6241278 [sys.catalog]: This master's current role is: LEADER
W20260925 04:44:35.748862 5089 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 66f5bc501a3443f3bce78dc1c6241278: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 04:44:35.748968 5089 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 04:44:35.749045 5083 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:44:35.749823 5083 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:44:35.756654 5083 catalog_manager.cc:1347] Generated new cluster ID: f3561581c6bc473ab9847b60f08e61f9
I20260925 04:44:35.756744 5083 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:44:35.779541 5083 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:44:35.779910 5083 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:44:35.786237 5083 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 66f5bc501a3443f3bce78dc1c6241278: Generated new TSK 0
I20260925 04:44:35.787070 5083 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:44:35.795311 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311449873301-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311449873301-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311449873301-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:0
--local_ip_for_outbound_sockets=127.0.98.65
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:37817
--builtin_ntp_servers=127.0.98.84:34513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 04:44:35.916486 5093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:44:35.916805 5093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:44:35.916893 5093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:44:35.921108 5093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:44:35.921196 5093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:44:35.921291 5093 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260925 04:44:35.926215 5093 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:34513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311449873301-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311449873301-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:37817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5093
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311449873301-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:44:35.927424 5093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:44:35.928853 5093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 04:44:35.939067 5093 server_base.cc:1034] running on GCE node
W20260925 04:44:35.939003 5101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:35.939023 5099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:35.939005 5098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:35.939688 5093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:44:35.940384 5093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:44:35.941570 5093 hybrid_clock.cc:648] HybridClock initialized: now 1790311475941555 us; error 39 us; skew 500 ppm
I20260925 04:44:35.943545 5093 webserver.cc:460] Webserver started at http://127.0.98.65:41293/ using document root <none> and password file <none>
I20260925 04:44:35.944232 5093 fs_manager.cc:359] Metadata directory not provided
I20260925 04:44:35.944329 5093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:44:35.944566 5093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:44:35.946924 5093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311449873301-393-0/minicluster-data/ts-0/data/instance:
uuid: "44960d4b1d914b8e88a10281db7bbaaf"
format_stamp: "Formatted at 2026-09-25 04:44:35 on dist-test-slave-kghz"
server_key: "6ac3222439f58b5e846dccd92affa6ac"
server_key_iv: "2517170facde650a5a087faeb9f496e5"
server_key_version: "encryptionkey@0"
I20260925 04:44:35.947499 5093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311449873301-393-0/minicluster-data/ts-0/wal/instance:
uuid: "44960d4b1d914b8e88a10281db7bbaaf"
format_stamp: "Formatted at 2026-09-25 04:44:35 on dist-test-slave-kghz"
server_key: "6ac3222439f58b5e846dccd92affa6ac"
server_key_iv: "2517170facde650a5a087faeb9f496e5"
server_key_version: "encryptionkey@0"
I20260925 04:44:35.951287 5093 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 04:44:35.953825 5107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:35.955128 5093 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 04:44:35.955272 5093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311449873301-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311449873301-393-0/minicluster-data/ts-0/wal
uuid: "44960d4b1d914b8e88a10281db7bbaaf"
format_stamp: "Formatted at 2026-09-25 04:44:35 on dist-test-slave-kghz"
server_key: "6ac3222439f58b5e846dccd92affa6ac"
server_key_iv: "2517170facde650a5a087faeb9f496e5"
server_key_version: "encryptionkey@0"
I20260925 04:44:35.955396 5093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311449873301-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311449873301-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311449873301-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:44:35.960127 5093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:44:35.960752 5093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:44:35.960943 5093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:44:35.961489 5093 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:44:35.962455 5093 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:44:35.962524 5093 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:35.962596 5093 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:44:35.962641 5093 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:35.971809 5093 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:34497
I20260925 04:44:35.971827 5220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:34497 every 8 connection(s)
I20260925 04:44:35.972738 5093 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
I20260925 04:44:35.979137 5072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:35.974687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:39333 (local address 127.0.98.126:37817)
0925 04:44:35.974862 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:35.974866 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:35.975892 (+ 1026us) server_negotiation.cc:407] Connection header received
0925 04:44:35.977302 (+ 1410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:35.977306 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:35.977352 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:35.977453 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:35.978639 (+ 1186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:35.978642 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:35.978645 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:44:35.978658 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:35.978679 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:44:35.978688 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:35.978793 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:35.978948 (+ 155us) server_negotiation.cc:299] Negotiation successful
0925 04:44:35.978995 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 04:44:35.980005 5223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:35.975052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:37817 (local address 127.0.98.65:39333)
0925 04:44:35.975730 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:35.975772 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:35.977079 (+ 1307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:35.977628 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:35.977635 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:35.977726 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:35.978377 (+ 651us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:44:35.978379 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:35.978421 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:44:35.978489 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:44:35.978785 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:44:35.978793 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:44:35.978798 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 04:44:35.978993 (+ 195us) client_negotiation.cc:240] Negotiation successful
0925 04:44:35.979244 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":106,"threads_started":1}
I20260925 04:44:35.981235 5221 heartbeater.cc:347] Connected to a master server at 127.0.98.126:37817
I20260925 04:44:35.981454 5221 heartbeater.cc:464] Registering TS with master...
I20260925 04:44:35.981839 5221 heartbeater.cc:511] Master 127.0.98.126:37817 requested a full tablet report, sending...
I20260925 04:44:35.982282 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 5093
I20260925 04:44:35.982409 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311449873301-393-0/minicluster-data/ts-0/wal/instance
I20260925 04:44:35.982698 393 external_mini_cluster.cc:1428] Setting key 40e9080e13dfa174ae47e6f300d58c86
I20260925 04:44:35.983565 5033 ts_manager.cc:194] Registered new tserver with Master: 44960d4b1d914b8e88a10281db7bbaaf (127.0.98.65:34497)
I20260925 04:44:35.984967 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311449873301-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311449873301-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311449873301-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:0
--local_ip_for_outbound_sockets=127.0.98.66
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:37817
--builtin_ntp_servers=127.0.98.84:34513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 04:44:36.103967 5224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:44:36.104310 5224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:44:36.104382 5224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:44:36.108690 5224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:44:36.108778 5224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:44:36.108872 5224 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260925 04:44:36.114071 5224 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:34513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311449873301-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311449873301-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:37817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5224
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311449873301-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:44:36.115310 5224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:44:36.116744 5224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:44:36.127296 5230 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:36.127315 5232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:36.127364 5224 server_base.cc:1034] running on GCE node
W20260925 04:44:36.127315 5229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:36.128124 5224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:44:36.128886 5224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:44:36.130100 5224 hybrid_clock.cc:648] HybridClock initialized: now 1790311476130056 us; error 66 us; skew 500 ppm
I20260925 04:44:36.132206 5224 webserver.cc:460] Webserver started at http://127.0.98.66:33469/ using document root <none> and password file <none>
I20260925 04:44:36.132834 5224 fs_manager.cc:359] Metadata directory not provided
I20260925 04:44:36.132934 5224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:44:36.133193 5224 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:44:36.135510 5224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311449873301-393-0/minicluster-data/ts-1/data/instance:
uuid: "14e2e76f143b4e008211e0beb92c1fee"
format_stamp: "Formatted at 2026-09-25 04:44:36 on dist-test-slave-kghz"
server_key: "109203d908f0bafd4b2ac043cfaa2f59"
server_key_iv: "bcdffcc6e82d2d8e56c1b625a80cb315"
server_key_version: "encryptionkey@0"
I20260925 04:44:36.136185 5224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311449873301-393-0/minicluster-data/ts-1/wal/instance:
uuid: "14e2e76f143b4e008211e0beb92c1fee"
format_stamp: "Formatted at 2026-09-25 04:44:36 on dist-test-slave-kghz"
server_key: "109203d908f0bafd4b2ac043cfaa2f59"
server_key_iv: "bcdffcc6e82d2d8e56c1b625a80cb315"
server_key_version: "encryptionkey@0"
I20260925 04:44:36.140360 5224 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 04:44:36.143131 5238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:36.144726 5224 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 04:44:36.144873 5224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311449873301-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311449873301-393-0/minicluster-data/ts-1/wal
uuid: "14e2e76f143b4e008211e0beb92c1fee"
format_stamp: "Formatted at 2026-09-25 04:44:36 on dist-test-slave-kghz"
server_key: "109203d908f0bafd4b2ac043cfaa2f59"
server_key_iv: "bcdffcc6e82d2d8e56c1b625a80cb315"
server_key_version: "encryptionkey@0"
I20260925 04:44:36.145005 5224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311449873301-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311449873301-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311449873301-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:44:36.149883 5224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:44:36.150676 5224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:44:36.150872 5224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:44:36.151454 5224 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:44:36.152408 5224 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:44:36.152480 5224 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:36.152555 5224 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:44:36.152603 5224 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:36.161618 5224 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:39841
I20260925 04:44:36.161782 5351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:39841 every 8 connection(s)
I20260925 04:44:36.162652 5224 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
I20260925 04:44:36.169333 5072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:36.164685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:45167 (local address 127.0.98.126:37817)
0925 04:44:36.164856 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:36.164860 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:36.166004 (+ 1144us) server_negotiation.cc:407] Connection header received
0925 04:44:36.167509 (+ 1505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:36.167513 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:36.167555 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:36.167655 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:36.168838 (+ 1183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:36.168841 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:36.168843 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:44:36.168856 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:36.168876 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:44:36.168886 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:36.168978 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:36.169167 (+ 189us) server_negotiation.cc:299] Negotiation successful
0925 04:44:36.169223 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 04:44:36.170217 5354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:36.165079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:37817 (local address 127.0.98.66:45167)
0925 04:44:36.165836 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:36.165880 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:36.167240 (+ 1360us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:36.167833 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:36.167841 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:36.167949 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:36.168558 (+ 609us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:44:36.168560 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:36.168603 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:44:36.168675 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:44:36.168980 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:44:36.168986 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:44:36.168991 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 04:44:36.169136 (+ 145us) client_negotiation.cc:240] Negotiation successful
0925 04:44:36.169340 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":110,"threads_started":1}
I20260925 04:44:36.171586 5352 heartbeater.cc:347] Connected to a master server at 127.0.98.126:37817
I20260925 04:44:36.171762 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 5224
I20260925 04:44:36.171882 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311449873301-393-0/minicluster-data/ts-1/wal/instance
I20260925 04:44:36.171859 5352 heartbeater.cc:464] Registering TS with master...
I20260925 04:44:36.172189 393 external_mini_cluster.cc:1428] Setting key 3ab829f322da90d76100ea69e5800573
I20260925 04:44:36.172415 5352 heartbeater.cc:511] Master 127.0.98.126:37817 requested a full tablet report, sending...
I20260925 04:44:36.173574 5032 ts_manager.cc:194] Registered new tserver with Master: 14e2e76f143b4e008211e0beb92c1fee (127.0.98.66:39841)
I20260925 04:44:36.174293 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311449873301-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311449873301-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311449873301-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:0
--local_ip_for_outbound_sockets=127.0.98.67
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:37817
--builtin_ntp_servers=127.0.98.84:34513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 04:44:36.292789 5355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:44:36.293092 5355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:44:36.293135 5355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:44:36.297529 5355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:44:36.297600 5355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:44:36.297657 5355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260925 04:44:36.302886 5355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:34513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311449873301-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311449873301-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:37817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5355
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311449873301-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:44:36.304039 5355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:44:36.305505 5355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:44:36.315716 5361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:36.315737 5363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:36.315786 5355 server_base.cc:1034] running on GCE node
W20260925 04:44:36.315742 5360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:36.316440 5355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:44:36.317044 5355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:44:36.318236 5355 hybrid_clock.cc:648] HybridClock initialized: now 1790311476318207 us; error 44 us; skew 500 ppm
I20260925 04:44:36.320504 5355 webserver.cc:460] Webserver started at http://127.0.98.67:40315/ using document root <none> and password file <none>
I20260925 04:44:36.321148 5355 fs_manager.cc:359] Metadata directory not provided
I20260925 04:44:36.321213 5355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:44:36.321383 5355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:44:36.323510 5355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311449873301-393-0/minicluster-data/ts-2/data/instance:
uuid: "6e760558aa5b4e2ea9de84c2696ef6d0"
format_stamp: "Formatted at 2026-09-25 04:44:36 on dist-test-slave-kghz"
server_key: "9b1b5ed8b97224a6fd22e2f94cae907d"
server_key_iv: "593673b32634bc20c4622e2c04666949"
server_key_version: "encryptionkey@0"
I20260925 04:44:36.324028 5355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311449873301-393-0/minicluster-data/ts-2/wal/instance:
uuid: "6e760558aa5b4e2ea9de84c2696ef6d0"
format_stamp: "Formatted at 2026-09-25 04:44:36 on dist-test-slave-kghz"
server_key: "9b1b5ed8b97224a6fd22e2f94cae907d"
server_key_iv: "593673b32634bc20c4622e2c04666949"
server_key_version: "encryptionkey@0"
I20260925 04:44:36.327912 5355 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 04:44:36.330626 5369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:36.332031 5355 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 04:44:36.332175 5355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311449873301-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311449873301-393-0/minicluster-data/ts-2/wal
uuid: "6e760558aa5b4e2ea9de84c2696ef6d0"
format_stamp: "Formatted at 2026-09-25 04:44:36 on dist-test-slave-kghz"
server_key: "9b1b5ed8b97224a6fd22e2f94cae907d"
server_key_iv: "593673b32634bc20c4622e2c04666949"
server_key_version: "encryptionkey@0"
I20260925 04:44:36.332276 5355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311449873301-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311449873301-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311449873301-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:44:36.337213 5355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:44:36.337811 5355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:44:36.337962 5355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:44:36.338585 5355 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:44:36.339460 5355 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:44:36.339506 5355 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:36.339545 5355 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:44:36.339578 5355 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:36.348685 5355 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:38255
I20260925 04:44:36.348708 5482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:38255 every 8 connection(s)
I20260925 04:44:36.349653 5355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
I20260925 04:44:36.351624 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 5355
I20260925 04:44:36.351737 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311449873301-393-0/minicluster-data/ts-2/wal/instance
I20260925 04:44:36.351986 393 external_mini_cluster.cc:1428] Setting key b13174f293580e8cd708c8d36684ba57
I20260925 04:44:36.356462 5072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:36.351565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:58621 (local address 127.0.98.126:37817)
0925 04:44:36.351736 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:36.351741 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:36.353124 (+ 1383us) server_negotiation.cc:407] Connection header received
0925 04:44:36.354684 (+ 1560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:36.354687 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:36.354730 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:36.354812 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:36.355979 (+ 1167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:36.355982 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:36.355985 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:44:36.355997 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:36.356018 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:44:36.356027 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:36.356128 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:36.356284 (+ 156us) server_negotiation.cc:299] Negotiation successful
0925 04:44:36.356329 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 04:44:36.357323 5485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:36.352137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:37817 (local address 127.0.98.67:58621)
0925 04:44:36.352979 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:36.353021 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:36.354474 (+ 1453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:36.354993 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:36.355000 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:36.355098 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:36.355720 (+ 622us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:44:36.355721 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:36.355762 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:44:36.355832 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:44:36.356128 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:44:36.356139 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:44:36.356147 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 04:44:36.356320 (+ 173us) client_negotiation.cc:240] Negotiation successful
0925 04:44:36.356565 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":132,"threads_started":1}
I20260925 04:44:36.358479 5483 heartbeater.cc:347] Connected to a master server at 127.0.98.126:37817
I20260925 04:44:36.358719 5483 heartbeater.cc:464] Registering TS with master...
I20260925 04:44:36.359189 5483 heartbeater.cc:511] Master 127.0.98.126:37817 requested a full tablet report, sending...
I20260925 04:44:36.360351 5032 ts_manager.cc:194] Registered new tserver with Master: 6e760558aa5b4e2ea9de84c2696ef6d0 (127.0.98.67:38255)
I20260925 04:44:36.366312 393 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 04:44:36.369967 5072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:36.368270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41196 (local address 127.0.98.126:37817)
0925 04:44:36.368424 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:36.368428 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:36.368636 (+ 208us) server_negotiation.cc:407] Connection header received
0925 04:44:36.368736 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:36.368738 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:36.368773 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:36.368839 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:36.369599 (+ 760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:36.369602 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:36.369604 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:44:36.369614 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:36.369633 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:44:36.369640 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:36.369710 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:36.369773 (+ 63us) server_negotiation.cc:299] Negotiation successful
0925 04:44:36.369804 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 04:44:36.373024 5032 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41196:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:44:36.376111 5032 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:44:36.388182 5498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:36.384899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:39841 (local address 127.0.0.1:57072)
0925 04:44:36.385999 (+ 1100us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:36.386015 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:36.386158 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:36.386618 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:36.386625 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:36.386704 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:36.387317 (+ 613us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:44:36.387318 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:36.387358 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:44:36.387430 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:44:36.387792 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:44:36.387801 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:44:36.387806 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 04:44:36.387921 (+ 115us) client_negotiation.cc:240] Negotiation successful
0925 04:44:36.388002 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1035,"mutex_wait_us":3,"thread_start_us":86,"threads_started":1}
I20260925 04:44:36.388191 5497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:36.385089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57072 (local address 127.0.98.66:39841)
0925 04:44:36.385711 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:36.385722 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:36.386095 (+ 373us) server_negotiation.cc:407] Connection header received
0925 04:44:36.386312 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:36.386318 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:36.386465 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:36.386612 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:36.387603 (+ 991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:36.387609 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:36.387628 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:44:36.387639 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:36.387681 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:44:36.387703 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:36.387819 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:36.387930 (+ 111us) server_negotiation.cc:299] Negotiation successful
0925 04:44:36.388015 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":87,"threads_started":1}
I20260925 04:44:36.388587 5496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:36.384937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46422 (local address 127.0.98.67:38255)
0925 04:44:36.385507 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:36.385521 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:36.385927 (+ 406us) server_negotiation.cc:407] Connection header received
0925 04:44:36.385989 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:36.385995 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:36.386200 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:36.386366 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:36.387601 (+ 1235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:36.387608 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:36.387621 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:44:36.387631 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:36.387671 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:44:36.387696 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:36.387808 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:36.388324 (+ 516us) server_negotiation.cc:299] Negotiation successful
0925 04:44:36.388399 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":190,"threads_started":1}
I20260925 04:44:36.388237 5495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:36.384830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:38255 (local address 127.0.0.1:46422)
0925 04:44:36.385671 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:36.385722 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:36.385823 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:36.387003 (+ 1180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:36.387006 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:36.387021 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:36.387434 (+ 413us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:44:36.387436 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:36.387444 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:44:36.387468 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:44:36.387857 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:44:36.387859 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:44:36.387862 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 04:44:36.387941 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 04:44:36.387993 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"thread_start_us":108,"threads_started":1}
I20260925 04:44:36.388967 5494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:36.385725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:34497 (local address 127.0.0.1:41524)
0925 04:44:36.386624 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:36.386640 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:36.386723 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:36.387853 (+ 1130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:36.387856 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:36.387871 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:36.388336 (+ 465us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:44:36.388338 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:36.388348 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:44:36.388373 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:44:36.388719 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:44:36.388722 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:44:36.388726 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 04:44:36.388794 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 04:44:36.388829 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":830,"thread_start_us":88,"threads_started":1}
I20260925 04:44:36.389420 5499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:36.386089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41524 (local address 127.0.98.65:34497)
0925 04:44:36.386501 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:36.386512 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:36.386835 (+ 323us) server_negotiation.cc:407] Connection header received
0925 04:44:36.386892 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:36.386898 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:36.387094 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:36.387230 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:36.388513 (+ 1283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:36.388519 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:36.388530 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:44:36.388542 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:36.388583 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:44:36.388621 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:36.388736 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:36.389064 (+ 328us) server_negotiation.cc:299] Negotiation successful
0925 04:44:36.389131 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":95,"threads_started":1}
I20260925 04:44:36.391494 5417 tablet_service.cc:1467] Processing CreateTablet for tablet bec6d2458cc64ef1bcc1c2bc4512d881 (DEFAULT_TABLE table=test-table [id=11e6f1d1e7694b0084664148c84f9065]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:44:36.391733 5155 tablet_service.cc:1467] Processing CreateTablet for tablet bec6d2458cc64ef1bcc1c2bc4512d881 (DEFAULT_TABLE table=test-table [id=11e6f1d1e7694b0084664148c84f9065]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:44:36.391733 5286 tablet_service.cc:1467] Processing CreateTablet for tablet bec6d2458cc64ef1bcc1c2bc4512d881 (DEFAULT_TABLE table=test-table [id=11e6f1d1e7694b0084664148c84f9065]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:44:36.392541 5417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bec6d2458cc64ef1bcc1c2bc4512d881. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:44:36.392725 5286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bec6d2458cc64ef1bcc1c2bc4512d881. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:44:36.392725 5155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bec6d2458cc64ef1bcc1c2bc4512d881. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:44:36.399153 5500 tablet_bootstrap.cc:492] T bec6d2458cc64ef1bcc1c2bc4512d881 P 6e760558aa5b4e2ea9de84c2696ef6d0: Bootstrap starting.
I20260925 04:44:36.400652 5502 tablet_bootstrap.cc:492] T bec6d2458cc64ef1bcc1c2bc4512d881 P 44960d4b1d914b8e88a10281db7bbaaf: Bootstrap starting.
I20260925 04:44:36.401494 5500 tablet_bootstrap.cc:654] T bec6d2458cc64ef1bcc1c2bc4512d881 P 6e760558aa5b4e2ea9de84c2696ef6d0: Neither blocks nor log segments found. Creating new log.
I20260925 04:44:36.402207 5501 tablet_bootstrap.cc:492] T bec6d2458cc64ef1bcc1c2bc4512d881 P 14e2e76f143b4e008211e0beb92c1fee: Bootstrap starting.
I20260925 04:44:36.402462 5500 log.cc:824] T bec6d2458cc64ef1bcc1c2bc4512d881 P 6e760558aa5b4e2ea9de84c2696ef6d0: Log is configured to *not* fsync() on all Append() calls
I20260925 04:44:36.402824 5502 tablet_bootstrap.cc:654] T bec6d2458cc64ef1bcc1c2bc4512d881 P 44960d4b1d914b8e88a10281db7bbaaf: Neither blocks nor log segments found. Creating new log.
I20260925 04:44:36.403838 5502 log.cc:824] T bec6d2458cc64ef1bcc1c2bc4512d881 P 44960d4b1d914b8e88a10281db7bbaaf: Log is configured to *not* fsync() on all Append() calls
I20260925 04:44:36.404410 5501 tablet_bootstrap.cc:654] T bec6d2458cc64ef1bcc1c2bc4512d881 P 14e2e76f143b4e008211e0beb92c1fee: Neither blocks nor log segments found. Creating new log.
I20260925 04:44:36.404771 5500 tablet_bootstrap.cc:492] T bec6d2458cc64ef1bcc1c2bc4512d881 P 6e760558aa5b4e2ea9de84c2696ef6d0: No bootstrap required, opened a new log
I20260925 04:44:36.405004 5500 ts_tablet_manager.cc:1397] T bec6d2458cc64ef1bcc1c2bc4512d881 P 6e760558aa5b4e2ea9de84c2696ef6d0: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260925 04:44:36.405280 5501 log.cc:824] T bec6d2458cc64ef1bcc1c2bc4512d881 P 14e2e76f143b4e008211e0beb92c1fee: Log is configured to *not* fsync() on all Append() calls
I20260925 04:44:36.406309 5502 tablet_bootstrap.cc:492] T bec6d2458cc64ef1bcc1c2bc4512d881 P 44960d4b1d914b8e88a10281db7bbaaf: No bootstrap required, opened a new log
I20260925 04:44:36.406497 5502 ts_tablet_manager.cc:1397] T bec6d2458cc64ef1bcc1c2bc4512d881 P 44960d4b1d914b8e88a10281db7bbaaf: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 04:44:36.407298 5501 tablet_bootstrap.cc:492] T bec6d2458cc64ef1bcc1c2bc4512d881 P 14e2e76f143b4e008211e0beb92c1fee: No bootstrap required, opened a new log
I20260925 04:44:36.407570 5501 ts_tablet_manager.cc:1397] T bec6d2458cc64ef1bcc1c2bc4512d881 P 14e2e76f143b4e008211e0beb92c1fee: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 04:44:36.409662 5500 raft_consensus.cc:348] T bec6d2458cc64ef1bcc1c2bc4512d881 P 6e760558aa5b4e2ea9de84c2696ef6d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e760558aa5b4e2ea9de84c2696ef6d0" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38255 } } peers { permanent_uuid: "14e2e76f143b4e008211e0beb92c1fee" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39841 } } peers { permanent_uuid: "44960d4b1d914b8e88a10281db7bbaaf" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34497 } }
I20260925 04:44:36.409936 5500 raft_consensus.cc:374] T bec6d2458cc64ef1bcc1c2bc4512d881 P 6e760558aa5b4e2ea9de84c2696ef6d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:44:36.410024 5500 raft_consensus.cc:729] T bec6d2458cc64ef1bcc1c2bc4512d881 P 6e760558aa5b4e2ea9de84c2696ef6d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e760558aa5b4e2ea9de84c2696ef6d0, State: Initialized, Role: FOLLOWER
I20260925 04:44:36.410395 5502 raft_consensus.cc:348] T bec6d2458cc64ef1bcc1c2bc4512d881 P 44960d4b1d914b8e88a10281db7bbaaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e760558aa5b4e2ea9de84c2696ef6d0" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38255 } } peers { permanent_uuid: "14e2e76f143b4e008211e0beb92c1fee" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39841 } } peers { permanent_uuid: "44960d4b1d914b8e88a10281db7bbaaf" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34497 } }
I20260925 04:44:36.410617 5502 raft_consensus.cc:374] T bec6d2458cc64ef1bcc1c2bc4512d881 P 44960d4b1d914b8e88a10281db7bbaaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:44:36.410589 5500 consensus_queue.cc:260] T bec6d2458cc64ef1bcc1c2bc4512d881 P 6e760558aa5b4e2ea9de84c2696ef6d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e760558aa5b4e2ea9de84c2696ef6d0" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38255 } } peers { permanent_uuid: "14e2e76f143b4e008211e0beb92c1fee" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39841 } } peers { permanent_uuid: "44960d4b1d914b8e88a10281db7bbaaf" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34497 } }
I20260925 04:44:36.410701 5502 raft_consensus.cc:729] T bec6d2458cc64ef1bcc1c2bc4512d881 P 44960d4b1d914b8e88a10281db7bbaaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44960d4b1d914b8e88a10281db7bbaaf, State: Initialized, Role: FOLLOWER
I20260925 04:44:36.411345 5483 heartbeater.cc:503] Master 127.0.98.126:37817 was elected leader, sending a full tablet report...
I20260925 04:44:36.411666 5500 ts_tablet_manager.cc:1428] T bec6d2458cc64ef1bcc1c2bc4512d881 P 6e760558aa5b4e2ea9de84c2696ef6d0: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260925 04:44:36.411173 5502 consensus_queue.cc:260] T bec6d2458cc64ef1bcc1c2bc4512d881 P 44960d4b1d914b8e88a10281db7bbaaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e760558aa5b4e2ea9de84c2696ef6d0" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38255 } } peers { permanent_uuid: "14e2e76f143b4e008211e0beb92c1fee" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39841 } } peers { permanent_uuid: "44960d4b1d914b8e88a10281db7bbaaf" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34497 } }
I20260925 04:44:36.412135 5501 raft_consensus.cc:348] T bec6d2458cc64ef1bcc1c2bc4512d881 P 14e2e76f143b4e008211e0beb92c1fee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e760558aa5b4e2ea9de84c2696ef6d0" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38255 } } peers { permanent_uuid: "14e2e76f143b4e008211e0beb92c1fee" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39841 } } peers { permanent_uuid: "44960d4b1d914b8e88a10281db7bbaaf" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34497 } }
I20260925 04:44:36.412388 5221 heartbeater.cc:503] Master 127.0.98.126:37817 was elected leader, sending a full tablet report...
I20260925 04:44:36.412431 5501 raft_consensus.cc:374] T bec6d2458cc64ef1bcc1c2bc4512d881 P 14e2e76f143b4e008211e0beb92c1fee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:44:36.412593 5501 raft_consensus.cc:729] T bec6d2458cc64ef1bcc1c2bc4512d881 P 14e2e76f143b4e008211e0beb92c1fee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14e2e76f143b4e008211e0beb92c1fee, State: Initialized, Role: FOLLOWER
I20260925 04:44:36.412729 5502 ts_tablet_manager.cc:1428] T bec6d2458cc64ef1bcc1c2bc4512d881 P 44960d4b1d914b8e88a10281db7bbaaf: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260925 04:44:36.413089 5501 consensus_queue.cc:260] T bec6d2458cc64ef1bcc1c2bc4512d881 P 14e2e76f143b4e008211e0beb92c1fee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e760558aa5b4e2ea9de84c2696ef6d0" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38255 } } peers { permanent_uuid: "14e2e76f143b4e008211e0beb92c1fee" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39841 } } peers { permanent_uuid: "44960d4b1d914b8e88a10281db7bbaaf" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34497 } }
I20260925 04:44:36.413967 5501 ts_tablet_manager.cc:1428] T bec6d2458cc64ef1bcc1c2bc4512d881 P 14e2e76f143b4e008211e0beb92c1fee: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260925 04:44:36.414127 5352 heartbeater.cc:503] Master 127.0.98.126:37817 was elected leader, sending a full tablet report...
W20260925 04:44:36.473956 5222 tablet.cc:2369] T bec6d2458cc64ef1bcc1c2bc4512d881 P 44960d4b1d914b8e88a10281db7bbaaf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:44:36.527436 5506 raft_consensus.cc:482] T bec6d2458cc64ef1bcc1c2bc4512d881 P 6e760558aa5b4e2ea9de84c2696ef6d0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:44:36.527647 5506 raft_consensus.cc:504] T bec6d2458cc64ef1bcc1c2bc4512d881 P 6e760558aa5b4e2ea9de84c2696ef6d0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e760558aa5b4e2ea9de84c2696ef6d0" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38255 } } peers { permanent_uuid: "14e2e76f143b4e008211e0beb92c1fee" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39841 } } peers { permanent_uuid: "44960d4b1d914b8e88a10281db7bbaaf" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34497 } }
I20260925 04:44:36.528764 5506 leader_election.cc:288] T bec6d2458cc64ef1bcc1c2bc4512d881 P 6e760558aa5b4e2ea9de84c2696ef6d0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 14e2e76f143b4e008211e0beb92c1fee (127.0.98.66:39841), 44960d4b1d914b8e88a10281db7bbaaf (127.0.98.65:34497)
I20260925 04:44:36.530814 5485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:36.529103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:34497 (local address 127.0.98.67:46965)
0925 04:44:36.529247 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:36.529266 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:36.529375 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:36.529644 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:36.529647 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:36.529665 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:36.530223 (+ 558us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:44:36.530225 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:36.530237 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:44:36.530282 (+ 45us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:44:36.530555 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:44:36.530558 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:44:36.530561 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 04:44:36.530646 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 04:44:36.530682 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 04:44:36.530815 5499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:36.529230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:46965 (local address 127.0.98.65:34497)
0925 04:44:36.529375 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:36.529381 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:36.529399 (+ 18us) server_negotiation.cc:407] Connection header received
0925 04:44:36.529483 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:36.529488 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:36.529525 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:36.529647 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:36.530410 (+ 763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:36.530414 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:36.530417 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:44:36.530429 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:36.530453 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:44:36.530467 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:36.530566 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:36.530641 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 04:44:36.530683 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 04:44:36.531394 5497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:36.529220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:43921 (local address 127.0.98.66:39841)
0925 04:44:36.529366 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:36.529374 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:36.529825 (+ 451us) server_negotiation.cc:407] Connection header received
0925 04:44:36.529976 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:36.529979 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:36.530025 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:36.530136 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:36.530949 (+ 813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:36.530953 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:36.530960 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:44:36.530977 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:36.531000 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:44:36.531014 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:36.531119 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:36.531226 (+ 107us) server_negotiation.cc:299] Negotiation successful
0925 04:44:36.531267 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 04:44:36.531420 5509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:36.529103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:39841 (local address 127.0.98.67:43921)
0925 04:44:36.529719 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:36.529732 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:36.529858 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:36.530152 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:36.530155 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:36.530171 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:36.530770 (+ 599us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:44:36.530773 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:36.530784 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:44:36.530818 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:44:36.531141 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:44:36.531145 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:44:36.531148 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 04:44:36.531229 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 04:44:36.531267 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"mutex_wait_us":181,"thread_start_us":201,"threads_started":1}
I20260925 04:44:36.531527 5175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bec6d2458cc64ef1bcc1c2bc4512d881" candidate_uuid: "6e760558aa5b4e2ea9de84c2696ef6d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44960d4b1d914b8e88a10281db7bbaaf" is_pre_election: true
I20260925 04:44:36.531908 5175 raft_consensus.cc:2454] T bec6d2458cc64ef1bcc1c2bc4512d881 P 44960d4b1d914b8e88a10281db7bbaaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e760558aa5b4e2ea9de84c2696ef6d0 in term 0.
I20260925 04:44:36.532254 5306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bec6d2458cc64ef1bcc1c2bc4512d881" candidate_uuid: "6e760558aa5b4e2ea9de84c2696ef6d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14e2e76f143b4e008211e0beb92c1fee" is_pre_election: true
I20260925 04:44:36.532440 5370 leader_election.cc:302] T bec6d2458cc64ef1bcc1c2bc4512d881 P 6e760558aa5b4e2ea9de84c2696ef6d0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44960d4b1d914b8e88a10281db7bbaaf, 6e760558aa5b4e2ea9de84c2696ef6d0; no voters:
I20260925 04:44:36.532536 5306 raft_consensus.cc:2454] T bec6d2458cc64ef1bcc1c2bc4512d881 P 14e2e76f143b4e008211e0beb92c1fee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e760558aa5b4e2ea9de84c2696ef6d0 in term 0.
I20260925 04:44:36.532750 5506 raft_consensus.cc:2781] T bec6d2458cc64ef1bcc1c2bc4512d881 P 6e760558aa5b4e2ea9de84c2696ef6d0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:44:36.532864 5506 raft_consensus.cc:482] T bec6d2458cc64ef1bcc1c2bc4512d881 P 6e760558aa5b4e2ea9de84c2696ef6d0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:44:36.532927 5506 raft_consensus.cc:3037] T bec6d2458cc64ef1bcc1c2bc4512d881 P 6e760558aa5b4e2ea9de84c2696ef6d0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:44:36.533978 5506 raft_consensus.cc:504] T bec6d2458cc64ef1bcc1c2bc4512d881 P 6e760558aa5b4e2ea9de84c2696ef6d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e760558aa5b4e2ea9de84c2696ef6d0" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38255 } } peers { permanent_uuid: "14e2e76f143b4e008211e0beb92c1fee" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39841 } } peers { permanent_uuid: "44960d4b1d914b8e88a10281db7bbaaf" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34497 } }
I20260925 04:44:36.534425 5506 leader_election.cc:288] T bec6d2458cc64ef1bcc1c2bc4512d881 P 6e760558aa5b4e2ea9de84c2696ef6d0 [CANDIDATE]: Term 1 election: Requested vote from peers 14e2e76f143b4e008211e0beb92c1fee (127.0.98.66:39841), 44960d4b1d914b8e88a10281db7bbaaf (127.0.98.65:34497)
I20260925 04:44:36.534768 5306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bec6d2458cc64ef1bcc1c2bc4512d881" candidate_uuid: "6e760558aa5b4e2ea9de84c2696ef6d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14e2e76f143b4e008211e0beb92c1fee"
I20260925 04:44:36.534808 5175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bec6d2458cc64ef1bcc1c2bc4512d881" candidate_uuid: "6e760558aa5b4e2ea9de84c2696ef6d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44960d4b1d914b8e88a10281db7bbaaf"
I20260925 04:44:36.534911 5306 raft_consensus.cc:3037] T bec6d2458cc64ef1bcc1c2bc4512d881 P 14e2e76f143b4e008211e0beb92c1fee [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:44:36.534938 5175 raft_consensus.cc:3037] T bec6d2458cc64ef1bcc1c2bc4512d881 P 44960d4b1d914b8e88a10281db7bbaaf [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:44:36.536121 5306 raft_consensus.cc:2454] T bec6d2458cc64ef1bcc1c2bc4512d881 P 14e2e76f143b4e008211e0beb92c1fee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e760558aa5b4e2ea9de84c2696ef6d0 in term 1.
I20260925 04:44:36.536193 5175 raft_consensus.cc:2454] T bec6d2458cc64ef1bcc1c2bc4512d881 P 44960d4b1d914b8e88a10281db7bbaaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e760558aa5b4e2ea9de84c2696ef6d0 in term 1.
I20260925 04:44:36.536458 5372 leader_election.cc:302] T bec6d2458cc64ef1bcc1c2bc4512d881 P 6e760558aa5b4e2ea9de84c2696ef6d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14e2e76f143b4e008211e0beb92c1fee, 6e760558aa5b4e2ea9de84c2696ef6d0; no voters:
I20260925 04:44:36.536717 5506 raft_consensus.cc:2781] T bec6d2458cc64ef1bcc1c2bc4512d881 P 6e760558aa5b4e2ea9de84c2696ef6d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:44:36.537082 5506 raft_consensus.cc:686] T bec6d2458cc64ef1bcc1c2bc4512d881 P 6e760558aa5b4e2ea9de84c2696ef6d0 [term 1 LEADER]: Becoming Leader. State: Replica: 6e760558aa5b4e2ea9de84c2696ef6d0, State: Running, Role: LEADER
I20260925 04:44:36.537479 5506 consensus_queue.cc:237] T bec6d2458cc64ef1bcc1c2bc4512d881 P 6e760558aa5b4e2ea9de84c2696ef6d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e760558aa5b4e2ea9de84c2696ef6d0" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38255 } } peers { permanent_uuid: "14e2e76f143b4e008211e0beb92c1fee" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39841 } } peers { permanent_uuid: "44960d4b1d914b8e88a10281db7bbaaf" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34497 } }
I20260925 04:44:36.541111 5031 catalog_manager.cc:5495] T bec6d2458cc64ef1bcc1c2bc4512d881 P 6e760558aa5b4e2ea9de84c2696ef6d0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e760558aa5b4e2ea9de84c2696ef6d0 (127.0.98.67). New cstate: current_term: 1 leader_uuid: "6e760558aa5b4e2ea9de84c2696ef6d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e760558aa5b4e2ea9de84c2696ef6d0" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38255 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14e2e76f143b4e008211e0beb92c1fee" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44960d4b1d914b8e88a10281db7bbaaf" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34497 } health_report { overall_health: UNKNOWN } } }
I20260925 04:44:36.580937 5496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:36.579282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46430 (local address 127.0.98.67:38255)
0925 04:44:36.579436 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:36.579441 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:36.579467 (+ 26us) server_negotiation.cc:407] Connection header received
0925 04:44:36.579568 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:36.579571 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:36.579633 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:36.579733 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:36.580482 (+ 749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:36.580485 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:36.580488 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:44:36.580499 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:36.580527 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:44:36.580535 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:36.580643 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:36.580739 (+ 96us) server_negotiation.cc:299] Negotiation successful
0925 04:44:36.580781 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 04:44:36.586947 5175 raft_consensus.cc:1261] T bec6d2458cc64ef1bcc1c2bc4512d881 P 44960d4b1d914b8e88a10281db7bbaaf [term 1 FOLLOWER]: Refusing update from remote peer 6e760558aa5b4e2ea9de84c2696ef6d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:44:36.586946 5306 raft_consensus.cc:1261] T bec6d2458cc64ef1bcc1c2bc4512d881 P 14e2e76f143b4e008211e0beb92c1fee [term 1 FOLLOWER]: Refusing update from remote peer 6e760558aa5b4e2ea9de84c2696ef6d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:44:36.587746 5506 consensus_queue.cc:1035] T bec6d2458cc64ef1bcc1c2bc4512d881 P 6e760558aa5b4e2ea9de84c2696ef6d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14e2e76f143b4e008211e0beb92c1fee" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:44:36.587996 5510 consensus_queue.cc:1035] T bec6d2458cc64ef1bcc1c2bc4512d881 P 6e760558aa5b4e2ea9de84c2696ef6d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44960d4b1d914b8e88a10281db7bbaaf" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:44:36.596098 5514 mvcc.cc:205] Tried to move back new op lower bound from 7333115808092839936 to 7333115807900893184. Current Snapshot: MvccSnapshot[applied={T|T < 7333115808092839936}]
I20260925 04:44:36.605388 5031 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41196:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:44:36.605611 5031 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41196:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:44:36.608233 5031 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 66f5bc501a3443f3bce78dc1c6241278: Sending DeleteTablet for 3 replicas of tablet bec6d2458cc64ef1bcc1c2bc4512d881
I20260925 04:44:36.609165 5417 tablet_service.cc:1514] Processing DeleteTablet for tablet bec6d2458cc64ef1bcc1c2bc4512d881 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:44:36 UTC) from {username='slave'} at 127.0.0.1:46422
I20260925 04:44:36.609169 5155 tablet_service.cc:1514] Processing DeleteTablet for tablet bec6d2458cc64ef1bcc1c2bc4512d881 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:44:36 UTC) from {username='slave'} at 127.0.0.1:41524
I20260925 04:44:36.609432 5286 tablet_service.cc:1514] Processing DeleteTablet for tablet bec6d2458cc64ef1bcc1c2bc4512d881 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:44:36 UTC) from {username='slave'} at 127.0.0.1:57072
I20260925 04:44:36.609800 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 5093
I20260925 04:44:36.609948 5523 tablet_replica.cc:330] T bec6d2458cc64ef1bcc1c2bc4512d881 P 6e760558aa5b4e2ea9de84c2696ef6d0: stopping tablet replica
I20260925 04:44:36.610430 5523 raft_consensus.cc:2229] T bec6d2458cc64ef1bcc1c2bc4512d881 P 6e760558aa5b4e2ea9de84c2696ef6d0 [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:44:36.610899 5523 raft_consensus.cc:2258] T bec6d2458cc64ef1bcc1c2bc4512d881 P 6e760558aa5b4e2ea9de84c2696ef6d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:44:36.611269 5525 tablet_replica.cc:330] T bec6d2458cc64ef1bcc1c2bc4512d881 P 14e2e76f143b4e008211e0beb92c1fee: stopping tablet replica
I20260925 04:44:36.611594 5525 raft_consensus.cc:2229] T bec6d2458cc64ef1bcc1c2bc4512d881 P 14e2e76f143b4e008211e0beb92c1fee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:44:36.611900 5525 raft_consensus.cc:2258] T bec6d2458cc64ef1bcc1c2bc4512d881 P 14e2e76f143b4e008211e0beb92c1fee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:44:36.613983 5525 ts_tablet_manager.cc:1905] T bec6d2458cc64ef1bcc1c2bc4512d881 P 14e2e76f143b4e008211e0beb92c1fee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:44:36.616976 5525 ts_tablet_manager.cc:1918] T bec6d2458cc64ef1bcc1c2bc4512d881 P 14e2e76f143b4e008211e0beb92c1fee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:44:36.617107 5525 log.cc:1196] T bec6d2458cc64ef1bcc1c2bc4512d881 P 14e2e76f143b4e008211e0beb92c1fee: Deleting WAL directory at /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311449873301-393-0/minicluster-data/ts-1/wal/wals/bec6d2458cc64ef1bcc1c2bc4512d881
I20260925 04:44:36.617483 5525 ts_tablet_manager.cc:1939] T bec6d2458cc64ef1bcc1c2bc4512d881 P 14e2e76f143b4e008211e0beb92c1fee: Deleting consensus metadata
W20260925 04:44:36.617942 5017 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.65:34497 (error 108)
I20260925 04:44:36.618902 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 5224
I20260925 04:44:36.618966 5019 catalog_manager.cc:4841] TS 14e2e76f143b4e008211e0beb92c1fee (127.0.98.66:39841): tablet bec6d2458cc64ef1bcc1c2bc4512d881 (table test-table [id=11e6f1d1e7694b0084664148c84f9065]) successfully deleted
I20260925 04:44:36.619462 5523 ts_tablet_manager.cc:1905] T bec6d2458cc64ef1bcc1c2bc4512d881 P 6e760558aa5b4e2ea9de84c2696ef6d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:44:36.621441 5494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:36.621016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:34497 (local address 127.0.0.1:41534)
0925 04:44:36.621228 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:36.621325 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.65:34497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
W20260925 04:44:36.621737 5017 catalog_manager.cc:4584] TS 44960d4b1d914b8e88a10281db7bbaaf (127.0.98.65:34497): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bec6d2458cc64ef1bcc1c2bc4512d881: Network error: Client connection negotiation failed: client connection to 127.0.98.65:34497: connect: Connection refused (error 111)
I20260925 04:44:36.622747 5523 ts_tablet_manager.cc:1918] T bec6d2458cc64ef1bcc1c2bc4512d881 P 6e760558aa5b4e2ea9de84c2696ef6d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:44:36.622889 5523 log.cc:1196] T bec6d2458cc64ef1bcc1c2bc4512d881 P 6e760558aa5b4e2ea9de84c2696ef6d0: Deleting WAL directory at /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790311449873301-393-0/minicluster-data/ts-2/wal/wals/bec6d2458cc64ef1bcc1c2bc4512d881
I20260925 04:44:36.623204 5523 ts_tablet_manager.cc:1939] T bec6d2458cc64ef1bcc1c2bc4512d881 P 6e760558aa5b4e2ea9de84c2696ef6d0: Deleting consensus metadata
I20260925 04:44:36.624413 5017 catalog_manager.cc:4841] TS 6e760558aa5b4e2ea9de84c2696ef6d0 (127.0.98.67:38255): tablet bec6d2458cc64ef1bcc1c2bc4512d881 (table test-table [id=11e6f1d1e7694b0084664148c84f9065]) successfully deleted
I20260925 04:44:36.626772 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 5355
I20260925 04:44:36.632035 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 5002
2026-09-25T04:44:36Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1117 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260925 04:44:36.654433 393 mini_oidc.cc:152] Starting JWKS server
I20260925 04:44:36.656430 393 webserver.cc:460] Webserver started at https://127.0.0.1:43355/ using document root <none> and password file <none>
I20260925 04:44:36.656621 393 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 04:44:36.656893 393 webserver.cc:460] Webserver started at http://127.0.0.1:45593/ using document root <none> and password file <none>
2026-09-25T04:44:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:44:36Z Disabled control of system clock
I20260925 04:44:36.666669 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790311449873301-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790311449873301-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790311449873301-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:38159
--webserver_interface=127.0.98.126
--webserver_port=0
--builtin_ntp_servers=127.0.98.84:46265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:38159
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43355/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790311449873301-393-0/testchainca.cert with env {}
W20260925 04:44:36.780283 5532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:44:36.780577 5532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:44:36.780619 5532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:44:36.784785 5532 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:44:36.784854 5532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:44:36.784869 5532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:44:36.784886 5532 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:44:36.784902 5532 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 04:44:36.784919 5532 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 04:44:36.784932 5532 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43355/jwks.json
I20260925 04:44:36.790436 5532 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:46265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790311449873301-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790311449873301-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:38159
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:38159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43355/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790311449873301-393-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5532
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790311449873301-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:44:36.791512 5532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:44:36.792579 5532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:44:36.801800 5538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:36.802022 5532 server_base.cc:1034] running on GCE node
W20260925 04:44:36.802037 5537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:36.801800 5540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:36.802800 5532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:44:36.803977 5532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:44:36.805189 5532 hybrid_clock.cc:648] HybridClock initialized: now 1790311476805178 us; error 39 us; skew 500 ppm
I20260925 04:44:36.807225 5532 webserver.cc:460] Webserver started at http://127.0.98.126:41609/ using document root <none> and password file <none>
I20260925 04:44:36.807799 5532 fs_manager.cc:359] Metadata directory not provided
I20260925 04:44:36.807891 5532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:44:36.808126 5532 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:44:36.810202 5532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790311449873301-393-0/minicluster-data/master-0/data/instance:
uuid: "78f0fcd57a074838a8514e4efd3f1179"
format_stamp: "Formatted at 2026-09-25 04:44:36 on dist-test-slave-kghz"
server_key: "7e766c7820050a11c5e7430ede8adac4"
server_key_iv: "ddec4e13f8cb567c3d628c75f9579072"
server_key_version: "encryptionkey@0"
I20260925 04:44:36.810824 5532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790311449873301-393-0/minicluster-data/master-0/wal/instance:
uuid: "78f0fcd57a074838a8514e4efd3f1179"
format_stamp: "Formatted at 2026-09-25 04:44:36 on dist-test-slave-kghz"
server_key: "7e766c7820050a11c5e7430ede8adac4"
server_key_iv: "ddec4e13f8cb567c3d628c75f9579072"
server_key_version: "encryptionkey@0"
I20260925 04:44:36.814586 5532 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:44:36.817068 5546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:36.818368 5532 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 04:44:36.818517 5532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790311449873301-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790311449873301-393-0/minicluster-data/master-0/wal
uuid: "78f0fcd57a074838a8514e4efd3f1179"
format_stamp: "Formatted at 2026-09-25 04:44:36 on dist-test-slave-kghz"
server_key: "7e766c7820050a11c5e7430ede8adac4"
server_key_iv: "ddec4e13f8cb567c3d628c75f9579072"
server_key_version: "encryptionkey@0"
I20260925 04:44:36.818642 5532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790311449873301-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790311449873301-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790311449873301-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:44:36.833806 5532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:44:36.834669 5532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:44:36.834882 5532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:44:36.841907 5532 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:38159
I20260925 04:44:36.841922 5598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:38159 every 8 connection(s)
I20260925 04:44:36.842988 5532 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
I20260925 04:44:36.846287 5599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:44:36.852100 5599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78f0fcd57a074838a8514e4efd3f1179: Bootstrap starting.
I20260925 04:44:36.853088 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 5532
I20260925 04:44:36.853266 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790311449873301-393-0/minicluster-data/master-0/wal/instance
I20260925 04:44:36.853580 393 external_mini_cluster.cc:1428] Setting key 545c46520a2f203befcd6924f4a0f0ee
I20260925 04:44:36.855059 5599 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 78f0fcd57a074838a8514e4efd3f1179: Neither blocks nor log segments found. Creating new log.
I20260925 04:44:36.855994 5599 log.cc:824] T 00000000000000000000000000000000 P 78f0fcd57a074838a8514e4efd3f1179: Log is configured to *not* fsync() on all Append() calls
I20260925 04:44:36.858543 5599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78f0fcd57a074838a8514e4efd3f1179: No bootstrap required, opened a new log
I20260925 04:44:36.863101 5599 raft_consensus.cc:348] T 00000000000000000000000000000000 P 78f0fcd57a074838a8514e4efd3f1179 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78f0fcd57a074838a8514e4efd3f1179" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 38159 } }
I20260925 04:44:36.863458 5599 raft_consensus.cc:374] T 00000000000000000000000000000000 P 78f0fcd57a074838a8514e4efd3f1179 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:44:36.863565 5599 raft_consensus.cc:729] T 00000000000000000000000000000000 P 78f0fcd57a074838a8514e4efd3f1179 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78f0fcd57a074838a8514e4efd3f1179, State: Initialized, Role: FOLLOWER
I20260925 04:44:36.864125 5599 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78f0fcd57a074838a8514e4efd3f1179 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78f0fcd57a074838a8514e4efd3f1179" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 38159 } }
I20260925 04:44:36.864305 5599 raft_consensus.cc:388] T 00000000000000000000000000000000 P 78f0fcd57a074838a8514e4efd3f1179 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:44:36.864372 5599 raft_consensus.cc:482] T 00000000000000000000000000000000 P 78f0fcd57a074838a8514e4efd3f1179 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:44:36.864471 5599 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 78f0fcd57a074838a8514e4efd3f1179 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:44:36.864849 5602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:36.854838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56402 (local address 127.0.98.126:38159)
0925 04:44:36.855711 (+ 873us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:36.855729 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:36.855777 (+ 48us) server_negotiation.cc:407] Connection header received
0925 04:44:36.856701 (+ 924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:36.856722 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:36.857181 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:36.857565 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:36.858013 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:36.859494 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:36.860469 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:36.861220 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:36.861399 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:36.862482 (+ 1083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:36.862489 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:36.862509 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:44:36.863044 (+ 535us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:36.863134 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:44:36.863162 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:36.863268 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:36.863453 (+ 185us) server_negotiation.cc:299] Negotiation successful
0925 04:44:36.863762 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":164,"threads_started":1}
I20260925 04:44:36.865604 5599 raft_consensus.cc:504] T 00000000000000000000000000000000 P 78f0fcd57a074838a8514e4efd3f1179 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78f0fcd57a074838a8514e4efd3f1179" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 38159 } }
I20260925 04:44:36.866003 5599 leader_election.cc:302] T 00000000000000000000000000000000 P 78f0fcd57a074838a8514e4efd3f1179 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 78f0fcd57a074838a8514e4efd3f1179; no voters:
I20260925 04:44:36.866386 5599 leader_election.cc:288] T 00000000000000000000000000000000 P 78f0fcd57a074838a8514e4efd3f1179 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:44:36.866536 5604 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 78f0fcd57a074838a8514e4efd3f1179 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:44:36.866832 5604 raft_consensus.cc:686] T 00000000000000000000000000000000 P 78f0fcd57a074838a8514e4efd3f1179 [term 1 LEADER]: Becoming Leader. State: Replica: 78f0fcd57a074838a8514e4efd3f1179, State: Running, Role: LEADER
I20260925 04:44:36.867600 5604 consensus_queue.cc:237] T 00000000000000000000000000000000 P 78f0fcd57a074838a8514e4efd3f1179 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78f0fcd57a074838a8514e4efd3f1179" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 38159 } }
I20260925 04:44:36.867789 5599 sys_catalog.cc:564] T 00000000000000000000000000000000 P 78f0fcd57a074838a8514e4efd3f1179 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:44:36.868196 393 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:44:36.869465 5605 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78f0fcd57a074838a8514e4efd3f1179 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "78f0fcd57a074838a8514e4efd3f1179" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78f0fcd57a074838a8514e4efd3f1179" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 38159 } } }
I20260925 04:44:36.869593 5605 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78f0fcd57a074838a8514e4efd3f1179 [sys.catalog]: This master's current role is: LEADER
I20260925 04:44:36.869901 5606 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78f0fcd57a074838a8514e4efd3f1179 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78f0fcd57a074838a8514e4efd3f1179. Latest consensus state: current_term: 1 leader_uuid: "78f0fcd57a074838a8514e4efd3f1179" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78f0fcd57a074838a8514e4efd3f1179" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 38159 } } }
I20260925 04:44:36.870018 5606 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78f0fcd57a074838a8514e4efd3f1179 [sys.catalog]: This master's current role is: LEADER
I20260925 04:44:36.870474 5615 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:44:36.873255 5615 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:44:36.879092 5615 catalog_manager.cc:1347] Generated new cluster ID: 31af335df8134e009a94b3624116b948
I20260925 04:44:36.879179 5615 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:44:36.908737 5615 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:44:36.910102 5615 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:44:36.923463 5615 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 78f0fcd57a074838a8514e4efd3f1179: Generated new TSK 0
I20260925 04:44:36.924315 5615 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 04:44:36.958055 5629 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260925 04:44:36.958182 5629 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 04:44:36.958698 5602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:36.933633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56410 (local address 127.0.98.126:38159)
0925 04:44:36.933822 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:36.933826 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:36.933965 (+ 139us) server_negotiation.cc:407] Connection header received
0925 04:44:36.934206 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:36.934209 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:36.934314 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:36.934401 (+ 87us) server_negotiation.cc:226] Negotiated authn=JWT
0925 04:44:36.934693 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:36.935228 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:36.936458 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:36.936675 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:36.936752 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:36.937222 (+ 470us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 04:44:36.958307 (+ 21085us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43355/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 04:44:36.958583 (+ 276us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56410: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43355/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":119}
W20260925 04:44:36.958808 5602 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56410: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43355/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260925 04:44:36.960142 5628 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:44:36.933483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38159 (local address 127.0.0.1:56410)
0925 04:44:36.933865 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:36.933881 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:36.934027 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:36.934443 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:36.934446 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:36.934455 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0925 04:44:36.934583 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:36.934590 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:36.935329 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:36.935332 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:36.936330 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:36.936340 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:36.936813 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:36.936817 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:36.936927 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:36.936985 (+ 58us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 04:44:36.958566 (+ 21581us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43355/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 04:44:36.958818 (+ 252us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.126:38159: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43355/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":124,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.126:38159: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43355/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260925 04:44:36.961068 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 5532
2026-09-25T04:44:36Z chronyd exiting
I20260925 04:44:36.998337 393 test_util.cc:175] -----------------------------------------------
I20260925 04:44:36.998425 393 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790311449873301-393-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (346 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260925 04:44:36.999496 393 mini_oidc.cc:152] Starting JWKS server
I20260925 04:44:37.000383 393 webserver.cc:460] Webserver started at https://127.0.0.1:38973/ using document root <none> and password file <none>
I20260925 04:44:37.000519 393 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 04:44:37.000784 393 webserver.cc:460] Webserver started at http://127.0.0.1:44199/ using document root <none> and password file <none>
2026-09-25T04:44:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:44:37Z Disabled control of system clock
I20260925 04:44:37.009743 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790311449873301-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790311449873301-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790311449873301-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:46743
--webserver_interface=127.0.98.126
--webserver_port=0
--builtin_ntp_servers=127.0.98.84:46775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:46743
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38973/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790311449873301-393-0/testchainca.cert with env {}
W20260925 04:44:37.121797 5635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:44:37.122114 5635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:44:37.122156 5635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:44:37.126870 5635 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:44:37.126957 5635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:44:37.126981 5635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:44:37.126999 5635 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:44:37.127015 5635 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 04:44:37.127032 5635 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 04:44:37.127045 5635 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38973/jwks.json
I20260925 04:44:37.133150 5635 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:46775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790311449873301-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790311449873301-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:46743
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:46743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38973/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790311449873301-393-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5635
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790311449873301-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:44:37.134420 5635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:44:37.135728 5635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:44:37.145507 5640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:37.145553 5635 server_base.cc:1034] running on GCE node
W20260925 04:44:37.145521 5641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:37.145521 5643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:37.146508 5635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:44:37.147745 5635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:44:37.148963 5635 hybrid_clock.cc:648] HybridClock initialized: now 1790311477148944 us; error 41 us; skew 500 ppm
I20260925 04:44:37.151224 5635 webserver.cc:460] Webserver started at http://127.0.98.126:43181/ using document root <none> and password file <none>
I20260925 04:44:37.151861 5635 fs_manager.cc:359] Metadata directory not provided
I20260925 04:44:37.151959 5635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:44:37.152200 5635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:44:37.154374 5635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790311449873301-393-0/minicluster-data/master-0/data/instance:
uuid: "18e09131a4d547b080c93d7a672a7c59"
format_stamp: "Formatted at 2026-09-25 04:44:37 on dist-test-slave-kghz"
server_key: "65acdfb04fb9bacff759f136dbc78f21"
server_key_iv: "a456e967d6018a8e23367155f28b1005"
server_key_version: "encryptionkey@0"
I20260925 04:44:37.154960 5635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790311449873301-393-0/minicluster-data/master-0/wal/instance:
uuid: "18e09131a4d547b080c93d7a672a7c59"
format_stamp: "Formatted at 2026-09-25 04:44:37 on dist-test-slave-kghz"
server_key: "65acdfb04fb9bacff759f136dbc78f21"
server_key_iv: "a456e967d6018a8e23367155f28b1005"
server_key_version: "encryptionkey@0"
I20260925 04:44:37.158906 5635 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 04:44:37.161696 5649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:37.163311 5635 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:44:37.163502 5635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790311449873301-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790311449873301-393-0/minicluster-data/master-0/wal
uuid: "18e09131a4d547b080c93d7a672a7c59"
format_stamp: "Formatted at 2026-09-25 04:44:37 on dist-test-slave-kghz"
server_key: "65acdfb04fb9bacff759f136dbc78f21"
server_key_iv: "a456e967d6018a8e23367155f28b1005"
server_key_version: "encryptionkey@0"
I20260925 04:44:37.163668 5635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790311449873301-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790311449873301-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790311449873301-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:44:37.198413 5635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:44:37.199188 5635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:44:37.199417 5635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:44:37.206987 5635 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:46743
I20260925 04:44:37.206998 5701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:46743 every 8 connection(s)
I20260925 04:44:37.207991 5635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
I20260925 04:44:37.211221 5702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:44:37.216526 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 5635
I20260925 04:44:37.216631 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790311449873301-393-0/minicluster-data/master-0/wal/instance
I20260925 04:44:37.216809 5702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18e09131a4d547b080c93d7a672a7c59: Bootstrap starting.
I20260925 04:44:37.216908 393 external_mini_cluster.cc:1428] Setting key 4f86f59a659390e5dd73db1cf1eda50b
I20260925 04:44:37.219729 5702 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 18e09131a4d547b080c93d7a672a7c59: Neither blocks nor log segments found. Creating new log.
I20260925 04:44:37.220544 5702 log.cc:824] T 00000000000000000000000000000000 P 18e09131a4d547b080c93d7a672a7c59: Log is configured to *not* fsync() on all Append() calls
I20260925 04:44:37.222571 5702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18e09131a4d547b080c93d7a672a7c59: No bootstrap required, opened a new log
I20260925 04:44:37.226686 5702 raft_consensus.cc:348] T 00000000000000000000000000000000 P 18e09131a4d547b080c93d7a672a7c59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18e09131a4d547b080c93d7a672a7c59" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46743 } }
I20260925 04:44:37.226883 5705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:37.218506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40784 (local address 127.0.98.126:46743)
0925 04:44:37.219215 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:37.219230 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:37.219267 (+ 37us) server_negotiation.cc:407] Connection header received
0925 04:44:37.219995 (+ 728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:37.220020 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:37.220476 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:37.220829 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:37.221263 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:37.222220 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:37.223230 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:37.223719 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:37.223888 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:37.224900 (+ 1012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:37.224906 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:37.224925 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:44:37.225369 (+ 444us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:37.225427 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:44:37.225447 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:37.225561 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:37.225709 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 04:44:37.226006 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":156,"threads_started":1}
I20260925 04:44:37.226974 5702 raft_consensus.cc:374] T 00000000000000000000000000000000 P 18e09131a4d547b080c93d7a672a7c59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:44:37.227075 5702 raft_consensus.cc:729] T 00000000000000000000000000000000 P 18e09131a4d547b080c93d7a672a7c59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18e09131a4d547b080c93d7a672a7c59, State: Initialized, Role: FOLLOWER
I20260925 04:44:37.227602 5702 consensus_queue.cc:260] T 00000000000000000000000000000000 P 18e09131a4d547b080c93d7a672a7c59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18e09131a4d547b080c93d7a672a7c59" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46743 } }
I20260925 04:44:37.227761 5702 raft_consensus.cc:388] T 00000000000000000000000000000000 P 18e09131a4d547b080c93d7a672a7c59 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:44:37.227814 5702 raft_consensus.cc:482] T 00000000000000000000000000000000 P 18e09131a4d547b080c93d7a672a7c59 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:44:37.227919 5702 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 18e09131a4d547b080c93d7a672a7c59 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:44:37.228925 5702 raft_consensus.cc:504] T 00000000000000000000000000000000 P 18e09131a4d547b080c93d7a672a7c59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18e09131a4d547b080c93d7a672a7c59" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46743 } }
I20260925 04:44:37.229307 5702 leader_election.cc:302] T 00000000000000000000000000000000 P 18e09131a4d547b080c93d7a672a7c59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 18e09131a4d547b080c93d7a672a7c59; no voters:
I20260925 04:44:37.229688 5702 leader_election.cc:288] T 00000000000000000000000000000000 P 18e09131a4d547b080c93d7a672a7c59 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:44:37.229825 5707 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 18e09131a4d547b080c93d7a672a7c59 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:44:37.230093 5707 raft_consensus.cc:686] T 00000000000000000000000000000000 P 18e09131a4d547b080c93d7a672a7c59 [term 1 LEADER]: Becoming Leader. State: Replica: 18e09131a4d547b080c93d7a672a7c59, State: Running, Role: LEADER
I20260925 04:44:37.230207 393 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:44:37.230554 5707 consensus_queue.cc:237] T 00000000000000000000000000000000 P 18e09131a4d547b080c93d7a672a7c59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18e09131a4d547b080c93d7a672a7c59" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46743 } }
I20260925 04:44:37.230947 5702 sys_catalog.cc:564] T 00000000000000000000000000000000 P 18e09131a4d547b080c93d7a672a7c59 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:44:37.232185 5708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18e09131a4d547b080c93d7a672a7c59 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "18e09131a4d547b080c93d7a672a7c59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18e09131a4d547b080c93d7a672a7c59" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46743 } } }
I20260925 04:44:37.232309 5708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18e09131a4d547b080c93d7a672a7c59 [sys.catalog]: This master's current role is: LEADER
I20260925 04:44:37.232775 5716 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:44:37.232734 5709 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18e09131a4d547b080c93d7a672a7c59 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 18e09131a4d547b080c93d7a672a7c59. Latest consensus state: current_term: 1 leader_uuid: "18e09131a4d547b080c93d7a672a7c59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18e09131a4d547b080c93d7a672a7c59" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46743 } } }
I20260925 04:44:37.232897 5709 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18e09131a4d547b080c93d7a672a7c59 [sys.catalog]: This master's current role is: LEADER
I20260925 04:44:37.235680 5716 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:44:37.241631 5716 catalog_manager.cc:1347] Generated new cluster ID: 74863fbfafb045f0b213f1215d43d2b2
I20260925 04:44:37.241791 5716 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:44:37.255278 5716 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:44:37.256247 5716 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:44:37.263180 5716 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 18e09131a4d547b080c93d7a672a7c59: Generated new TSK 0
I20260925 04:44:37.263901 5716 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 04:44:37.297405 5732 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260925 04:44:37.297508 5732 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 04:44:37.298163 5705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:37.274526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40796 (local address 127.0.98.126:46743)
0925 04:44:37.274688 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:37.274694 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:37.274854 (+ 160us) server_negotiation.cc:407] Connection header received
0925 04:44:37.275024 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:37.275027 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:37.275077 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:37.275182 (+ 105us) server_negotiation.cc:226] Negotiated authn=JWT
0925 04:44:37.275430 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:37.275980 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:37.277274 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:37.277498 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:37.277592 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:37.277923 (+ 331us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 04:44:37.297776 (+ 19853us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38973/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 04:44:37.298037 (+ 261us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40796: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38973/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":54}
W20260925 04:44:37.298254 5705 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40796: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38973/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260925 04:44:37.298404 5731 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:44:37.274461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:46743 (local address 127.0.0.1:40796)
0925 04:44:37.274756 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:37.274772 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:37.274896 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:37.275205 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:37.275208 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:37.275216 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0925 04:44:37.275328 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:37.275334 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:37.276091 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:37.276095 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:37.277106 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:37.277116 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:37.277622 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:37.277626 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:37.277714 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:37.277744 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 04:44:37.298008 (+ 20264us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38973/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 04:44:37.298200 (+ 192us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.126:46743: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38973/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":115,"threads_started":1}
I20260925 04:44:37.299218 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 5635
2026-09-25T04:44:37Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (339 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260925 04:44:37.339098 393 mini_oidc.cc:152] Starting JWKS server
I20260925 04:44:37.340018 393 webserver.cc:460] Webserver started at https://127.0.0.1:33583/ using document root <none> and password file <none>
I20260925 04:44:37.340104 393 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 04:44:37.340384 393 webserver.cc:460] Webserver started at http://127.0.0.1:33285/ using document root <none> and password file <none>
2026-09-25T04:44:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:44:37Z Disabled control of system clock
I20260925 04:44:37.349344 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790311449873301-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790311449873301-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790311449873301-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:35673
--webserver_interface=127.0.98.126
--webserver_port=0
--builtin_ntp_servers=127.0.98.84:36051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:35673
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33583/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 04:44:37.463653 5738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:44:37.464073 5738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:44:37.464160 5738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:44:37.468389 5738 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:44:37.468509 5738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:44:37.468537 5738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:44:37.468577 5738 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:44:37.468616 5738 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 04:44:37.468649 5738 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 04:44:37.468683 5738 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33583/jwks.json
I20260925 04:44:37.473747 5738 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:36051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790311449873301-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790311449873301-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:35673
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:35673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33583/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5738
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790311449873301-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:44:37.475143 5738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:44:37.476581 5738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:44:37.485852 5744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:37.485883 5746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:37.485847 5743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:37.486522 5738 server_base.cc:1034] running on GCE node
I20260925 04:44:37.487186 5738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:44:37.488354 5738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:44:37.489565 5738 hybrid_clock.cc:648] HybridClock initialized: now 1790311477489554 us; error 34 us; skew 500 ppm
I20260925 04:44:37.491570 5738 webserver.cc:460] Webserver started at http://127.0.98.126:42955/ using document root <none> and password file <none>
I20260925 04:44:37.492182 5738 fs_manager.cc:359] Metadata directory not provided
I20260925 04:44:37.492273 5738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:44:37.492522 5738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:44:37.494673 5738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790311449873301-393-0/minicluster-data/master-0/data/instance:
uuid: "87c9a5a0e26f44c493943a4dc6013481"
format_stamp: "Formatted at 2026-09-25 04:44:37 on dist-test-slave-kghz"
server_key: "0514349501ff496a3a08741d3efef570"
server_key_iv: "cdf829a4b6fbc313aaf72f2ef4450e7d"
server_key_version: "encryptionkey@0"
I20260925 04:44:37.495289 5738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790311449873301-393-0/minicluster-data/master-0/wal/instance:
uuid: "87c9a5a0e26f44c493943a4dc6013481"
format_stamp: "Formatted at 2026-09-25 04:44:37 on dist-test-slave-kghz"
server_key: "0514349501ff496a3a08741d3efef570"
server_key_iv: "cdf829a4b6fbc313aaf72f2ef4450e7d"
server_key_version: "encryptionkey@0"
I20260925 04:44:37.499109 5738 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 04:44:37.501737 5752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:37.503039 5738 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260925 04:44:37.503193 5738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790311449873301-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790311449873301-393-0/minicluster-data/master-0/wal
uuid: "87c9a5a0e26f44c493943a4dc6013481"
format_stamp: "Formatted at 2026-09-25 04:44:37 on dist-test-slave-kghz"
server_key: "0514349501ff496a3a08741d3efef570"
server_key_iv: "cdf829a4b6fbc313aaf72f2ef4450e7d"
server_key_version: "encryptionkey@0"
I20260925 04:44:37.503320 5738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790311449873301-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790311449873301-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790311449873301-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:44:37.522734 5738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:44:37.523507 5738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:44:37.523713 5738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:44:37.530874 5738 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:35673
I20260925 04:44:37.530889 5804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:35673 every 8 connection(s)
I20260925 04:44:37.531855 5738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
I20260925 04:44:37.535378 5805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:44:37.535755 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 5738
I20260925 04:44:37.535876 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790311449873301-393-0/minicluster-data/master-0/wal/instance
I20260925 04:44:37.536155 393 external_mini_cluster.cc:1428] Setting key 2f3e1ebf2bd5634010225e3714d4df5a
I20260925 04:44:37.542989 5805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87c9a5a0e26f44c493943a4dc6013481: Bootstrap starting.
I20260925 04:44:37.545653 5805 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 87c9a5a0e26f44c493943a4dc6013481: Neither blocks nor log segments found. Creating new log.
I20260925 04:44:37.546352 5808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:37.537423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53442 (local address 127.0.98.126:35673)
0925 04:44:37.538055 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:37.538070 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:37.538107 (+ 37us) server_negotiation.cc:407] Connection header received
0925 04:44:37.538852 (+ 745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:37.538871 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:37.539311 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:37.539605 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:37.540078 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:37.541166 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:37.542543 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:37.543036 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:37.543238 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:37.544401 (+ 1163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:37.544406 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:37.544419 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:44:37.544819 (+ 400us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:37.544876 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:44:37.544900 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:37.545014 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:37.545179 (+ 165us) server_negotiation.cc:299] Negotiation successful
0925 04:44:37.545471 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":138,"threads_started":1}
I20260925 04:44:37.546654 5805 log.cc:824] T 00000000000000000000000000000000 P 87c9a5a0e26f44c493943a4dc6013481: Log is configured to *not* fsync() on all Append() calls
I20260925 04:44:37.548902 5805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87c9a5a0e26f44c493943a4dc6013481: No bootstrap required, opened a new log
I20260925 04:44:37.549198 393 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:44:37.552953 5805 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87c9a5a0e26f44c493943a4dc6013481 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87c9a5a0e26f44c493943a4dc6013481" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 35673 } }
I20260925 04:44:37.553243 5805 raft_consensus.cc:374] T 00000000000000000000000000000000 P 87c9a5a0e26f44c493943a4dc6013481 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:44:37.553356 5805 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87c9a5a0e26f44c493943a4dc6013481 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87c9a5a0e26f44c493943a4dc6013481, State: Initialized, Role: FOLLOWER
I20260925 04:44:37.553828 5805 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87c9a5a0e26f44c493943a4dc6013481 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87c9a5a0e26f44c493943a4dc6013481" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 35673 } }
I20260925 04:44:37.553993 5805 raft_consensus.cc:388] T 00000000000000000000000000000000 P 87c9a5a0e26f44c493943a4dc6013481 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:44:37.554106 5805 raft_consensus.cc:482] T 00000000000000000000000000000000 P 87c9a5a0e26f44c493943a4dc6013481 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:44:37.554239 5805 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87c9a5a0e26f44c493943a4dc6013481 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:44:37.555346 5805 raft_consensus.cc:504] T 00000000000000000000000000000000 P 87c9a5a0e26f44c493943a4dc6013481 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87c9a5a0e26f44c493943a4dc6013481" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 35673 } }
I20260925 04:44:37.555837 5805 leader_election.cc:302] T 00000000000000000000000000000000 P 87c9a5a0e26f44c493943a4dc6013481 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 87c9a5a0e26f44c493943a4dc6013481; no voters:
I20260925 04:44:37.556288 5805 leader_election.cc:288] T 00000000000000000000000000000000 P 87c9a5a0e26f44c493943a4dc6013481 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:44:37.556712 5811 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 87c9a5a0e26f44c493943a4dc6013481 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:44:37.556973 5811 raft_consensus.cc:686] T 00000000000000000000000000000000 P 87c9a5a0e26f44c493943a4dc6013481 [term 1 LEADER]: Becoming Leader. State: Replica: 87c9a5a0e26f44c493943a4dc6013481, State: Running, Role: LEADER
I20260925 04:44:37.557389 5811 consensus_queue.cc:237] T 00000000000000000000000000000000 P 87c9a5a0e26f44c493943a4dc6013481 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87c9a5a0e26f44c493943a4dc6013481" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 35673 } }
I20260925 04:44:37.557809 5805 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87c9a5a0e26f44c493943a4dc6013481 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:44:37.558920 5811 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87c9a5a0e26f44c493943a4dc6013481 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87c9a5a0e26f44c493943a4dc6013481. Latest consensus state: current_term: 1 leader_uuid: "87c9a5a0e26f44c493943a4dc6013481" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87c9a5a0e26f44c493943a4dc6013481" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 35673 } } }
I20260925 04:44:37.559118 5811 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87c9a5a0e26f44c493943a4dc6013481 [sys.catalog]: This master's current role is: LEADER
I20260925 04:44:37.559383 5812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87c9a5a0e26f44c493943a4dc6013481 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "87c9a5a0e26f44c493943a4dc6013481" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87c9a5a0e26f44c493943a4dc6013481" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 35673 } } }
I20260925 04:44:37.559495 5812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87c9a5a0e26f44c493943a4dc6013481 [sys.catalog]: This master's current role is: LEADER
I20260925 04:44:37.559568 5819 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:44:37.562912 5819 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:44:37.568707 5819 catalog_manager.cc:1347] Generated new cluster ID: 7e95d3d878b443ab8709bb1cbc2a2d7c
I20260925 04:44:37.568848 5819 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:44:37.597404 5819 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:44:37.598743 5819 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:44:37.609401 5819 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 87c9a5a0e26f44c493943a4dc6013481: Generated new TSK 0
I20260925 04:44:37.610335 5819 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:44:37.628333 5808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:37.623073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53450 (local address 127.0.98.126:35673)
0925 04:44:37.623233 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:37.623241 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:37.623433 (+ 192us) server_negotiation.cc:407] Connection header received
0925 04:44:37.623638 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:37.623641 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:37.623694 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:37.623801 (+ 107us) server_negotiation.cc:226] Negotiated authn=JWT
0925 04:44:37.624104 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:37.624640 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:37.625911 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:37.626139 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:37.626226 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:37.626625 (+ 399us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 04:44:37.627930 (+ 1305us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33583/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 04:44:37.628202 (+ 272us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53450: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33583/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":68}
W20260925 04:44:37.628433 5808 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53450: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33583/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260925 04:44:37.631439 5834 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:44:37.622969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:35673 (local address 127.0.0.1:53450)
0925 04:44:37.623329 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:37.623347 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:37.623496 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:37.623838 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:37.623841 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:37.623852 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0925 04:44:37.623977 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:37.623986 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:37.624764 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:37.624768 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:37.625773 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:37.625785 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:37.626294 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:37.626298 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:37.626399 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:37.626430 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 04:44:37.628129 (+ 1699us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33583/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 04:44:37.631268 (+ 3139us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.126:35673: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33583/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":266,"mutex_wait_us":2810,"thread_start_us":138,"threads_started":1}
I20260925 04:44:37.632253 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 5738
E20260925 04:44:37.640421 5835 webserver.cc:533] Webserver: sslize failed:
E20260925 04:44:37.640538 5835 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-25T04:44:37Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (333 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-25T04:44:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:44:37Z Disabled control of system clock
I20260925 04:44:37.681619 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790311449873301-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790311449873301-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790311449873301-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:34399
--webserver_interface=127.0.98.126
--webserver_port=0
--builtin_ntp_servers=127.0.98.84:41509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:34399
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790311449873301-393-0/testchainca.cert with env {}
W20260925 04:44:37.799492 5839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:44:37.799789 5839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:44:37.799832 5839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:44:37.804108 5839 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:44:37.804183 5839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:44:37.804198 5839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:44:37.804215 5839 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:44:37.804231 5839 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 04:44:37.804247 5839 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 04:44:37.804302 5839 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260925 04:44:37.810006 5839 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:41509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790311449873301-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790311449873301-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:34399
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:34399
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790311449873301-393-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790311449873301-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:44:37.811157 5839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:44:37.812175 5839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:44:37.821213 5845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:37.821255 5839 server_base.cc:1034] running on GCE node
W20260925 04:44:37.821200 5844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:37.821195 5847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:37.822053 5839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:44:37.823258 5839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:44:37.824434 5839 hybrid_clock.cc:648] HybridClock initialized: now 1790311477824413 us; error 36 us; skew 500 ppm
I20260925 04:44:37.826613 5839 webserver.cc:460] Webserver started at http://127.0.98.126:42261/ using document root <none> and password file <none>
I20260925 04:44:37.827246 5839 fs_manager.cc:359] Metadata directory not provided
I20260925 04:44:37.827320 5839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:44:37.827558 5839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:44:37.829782 5839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790311449873301-393-0/minicluster-data/master-0/data/instance:
uuid: "3e4d16df7bbf4a4d93ee65f19b59c65f"
format_stamp: "Formatted at 2026-09-25 04:44:37 on dist-test-slave-kghz"
server_key: "3c7944c6f40c626c23c85ccd185f9af9"
server_key_iv: "8834bfbfa72fcc1a5875cfd8f5933576"
server_key_version: "encryptionkey@0"
I20260925 04:44:37.830425 5839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790311449873301-393-0/minicluster-data/master-0/wal/instance:
uuid: "3e4d16df7bbf4a4d93ee65f19b59c65f"
format_stamp: "Formatted at 2026-09-25 04:44:37 on dist-test-slave-kghz"
server_key: "3c7944c6f40c626c23c85ccd185f9af9"
server_key_iv: "8834bfbfa72fcc1a5875cfd8f5933576"
server_key_version: "encryptionkey@0"
I20260925 04:44:37.834509 5839 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 04:44:37.837260 5853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:37.838655 5839 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:44:37.838811 5839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790311449873301-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790311449873301-393-0/minicluster-data/master-0/wal
uuid: "3e4d16df7bbf4a4d93ee65f19b59c65f"
format_stamp: "Formatted at 2026-09-25 04:44:37 on dist-test-slave-kghz"
server_key: "3c7944c6f40c626c23c85ccd185f9af9"
server_key_iv: "8834bfbfa72fcc1a5875cfd8f5933576"
server_key_version: "encryptionkey@0"
I20260925 04:44:37.838940 5839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790311449873301-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790311449873301-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790311449873301-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:44:37.853516 5839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:44:37.854351 5839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:44:37.854548 5839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:44:37.861708 5839 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:34399
I20260925 04:44:37.861758 5905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:34399 every 8 connection(s)
I20260925 04:44:37.862764 5839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
I20260925 04:44:37.866736 5906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:44:37.868424 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 5839
I20260925 04:44:37.868551 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790311449873301-393-0/minicluster-data/master-0/wal/instance
I20260925 04:44:37.868844 393 external_mini_cluster.cc:1428] Setting key 16536eecde26484609e276e73275b0d3
I20260925 04:44:37.873524 5906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e4d16df7bbf4a4d93ee65f19b59c65f: Bootstrap starting.
I20260925 04:44:37.876596 5906 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e4d16df7bbf4a4d93ee65f19b59c65f: Neither blocks nor log segments found. Creating new log.
I20260925 04:44:37.877545 5906 log.cc:824] T 00000000000000000000000000000000 P 3e4d16df7bbf4a4d93ee65f19b59c65f: Log is configured to *not* fsync() on all Append() calls
I20260925 04:44:37.880151 5906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e4d16df7bbf4a4d93ee65f19b59c65f: No bootstrap required, opened a new log
I20260925 04:44:37.883011 5909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:37.870961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43750 (local address 127.0.98.126:34399)
0925 04:44:37.873367 (+ 2406us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:37.873389 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:37.873438 (+ 49us) server_negotiation.cc:407] Connection header received
0925 04:44:37.874410 (+ 972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:37.874440 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:37.875039 (+ 599us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:37.875554 (+ 515us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:37.876181 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:37.877759 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:37.879051 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:37.879546 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:37.879673 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:37.880709 (+ 1036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:37.880715 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:37.880728 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:44:37.881176 (+ 448us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:37.881237 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:44:37.881258 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:37.881345 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:37.881540 (+ 195us) server_negotiation.cc:299] Negotiation successful
0925 04:44:37.881861 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1904,"thread_start_us":197,"threads_started":1}
I20260925 04:44:37.883728 5906 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e4d16df7bbf4a4d93ee65f19b59c65f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e4d16df7bbf4a4d93ee65f19b59c65f" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34399 } }
I20260925 04:44:37.883998 5906 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e4d16df7bbf4a4d93ee65f19b59c65f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:44:37.884080 5906 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e4d16df7bbf4a4d93ee65f19b59c65f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e4d16df7bbf4a4d93ee65f19b59c65f, State: Initialized, Role: FOLLOWER
I20260925 04:44:37.884541 5906 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e4d16df7bbf4a4d93ee65f19b59c65f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e4d16df7bbf4a4d93ee65f19b59c65f" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34399 } }
I20260925 04:44:37.884701 5906 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e4d16df7bbf4a4d93ee65f19b59c65f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:44:37.884788 5906 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e4d16df7bbf4a4d93ee65f19b59c65f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:44:37.884908 5906 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e4d16df7bbf4a4d93ee65f19b59c65f [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:44:37.886353 393 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:44:37.940222 5906 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e4d16df7bbf4a4d93ee65f19b59c65f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e4d16df7bbf4a4d93ee65f19b59c65f" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34399 } }
I20260925 04:44:37.940831 5906 leader_election.cc:302] T 00000000000000000000000000000000 P 3e4d16df7bbf4a4d93ee65f19b59c65f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e4d16df7bbf4a4d93ee65f19b59c65f; no voters:
I20260925 04:44:37.941342 5906 leader_election.cc:288] T 00000000000000000000000000000000 P 3e4d16df7bbf4a4d93ee65f19b59c65f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:44:37.941475 5911 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e4d16df7bbf4a4d93ee65f19b59c65f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:44:37.941790 5911 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e4d16df7bbf4a4d93ee65f19b59c65f [term 1 LEADER]: Becoming Leader. State: Replica: 3e4d16df7bbf4a4d93ee65f19b59c65f, State: Running, Role: LEADER
I20260925 04:44:37.942292 5911 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e4d16df7bbf4a4d93ee65f19b59c65f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e4d16df7bbf4a4d93ee65f19b59c65f" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34399 } }
I20260925 04:44:37.943078 5906 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e4d16df7bbf4a4d93ee65f19b59c65f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:44:37.944303 5912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e4d16df7bbf4a4d93ee65f19b59c65f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e4d16df7bbf4a4d93ee65f19b59c65f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e4d16df7bbf4a4d93ee65f19b59c65f" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34399 } } }
I20260925 04:44:37.944473 5912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e4d16df7bbf4a4d93ee65f19b59c65f [sys.catalog]: This master's current role is: LEADER
I20260925 04:44:37.944736 5913 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e4d16df7bbf4a4d93ee65f19b59c65f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e4d16df7bbf4a4d93ee65f19b59c65f. Latest consensus state: current_term: 1 leader_uuid: "3e4d16df7bbf4a4d93ee65f19b59c65f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e4d16df7bbf4a4d93ee65f19b59c65f" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34399 } } }
I20260925 04:44:37.944914 5913 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e4d16df7bbf4a4d93ee65f19b59c65f [sys.catalog]: This master's current role is: LEADER
W20260925 04:44:37.948063 5926 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3e4d16df7bbf4a4d93ee65f19b59c65f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 04:44:37.948153 5926 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 04:44:37.948242 5920 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:44:37.949055 5920 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:44:37.954754 5920 catalog_manager.cc:1347] Generated new cluster ID: 82b960facdfc4cd2bd11ce39a8a92b1c
I20260925 04:44:37.954836 5920 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:44:37.973830 5920 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:44:37.975333 5920 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:44:37.985754 5920 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e4d16df7bbf4a4d93ee65f19b59c65f: Generated new TSK 0
I20260925 04:44:37.986768 5920 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:44:38.060094 5909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:38.001861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43756 (local address 127.0.98.126:34399)
0925 04:44:38.002032 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:38.002038 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:38.002219 (+ 181us) server_negotiation.cc:407] Connection header received
0925 04:44:38.002443 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:38.002446 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:38.002498 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:38.002588 (+ 90us) server_negotiation.cc:226] Negotiated authn=JWT
0925 04:44:38.002877 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:38.003413 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:38.004669 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:38.004890 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:38.004973 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:38.005314 (+ 341us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 04:44:38.059705 (+ 54391us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 04:44:38.059976 (+ 271us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43756: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":76}
W20260925 04:44:38.060245 5909 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43756: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260925 04:44:38.060433 5935 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:44:38.001769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34399 (local address 127.0.0.1:43756)
0925 04:44:38.002108 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:38.002125 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:38.002285 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:38.002618 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:38.002621 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:38.002630 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0925 04:44:38.002755 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:38.002762 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:38.003528 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:38.003532 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:38.004539 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:38.004550 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:38.005012 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:38.005016 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:38.005111 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:38.005141 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 04:44:38.059983 (+ 54842us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 04:44:38.060219 (+ 236us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.126:34399: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":110,"threads_started":1}
I20260925 04:44:38.061069 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 5839
2026-09-25T04:44:38Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (407 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (134 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (128 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:44:38 dist-test-slave-kghz krb5kdc[5947](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:44:38 dist-test-slave-kghz krb5kdc[5947](info): set up 2 sockets
Sep 25 04:44:38 dist-test-slave-kghz krb5kdc[5947](info): commencing operation
krb5kdc: starting...
W20260925 04:44:39.898765 393 mini_kdc.cc:121] Time spent starting KDC: real 1.555s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/krb5kdc/test-user.keytab.
Sep 25 04:44:39 dist-test-slave-kghz krb5kdc[5947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:44:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:44:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260925 04:44:40.026494 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:34095
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:40049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:34095
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:44:40.139174 5960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:44:40.139479 5960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:44:40.139523 5960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:44:40.143739 5960 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:44:40.143810 5960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:44:40.143831 5960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:44:40.143847 5960 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:44:40.143863 5960 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:44:40.149478 5960 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:40049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:34095
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:34095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:44:40.150610 5960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:44:40.151593 5960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:44:40.160302 5965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:40.160312 5968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:40.160321 5966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:40.160372 5960 server_base.cc:1034] running on GCE node
I20260925 04:44:40.161140 5960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:44:40.162290 5960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:44:40.163517 5960 hybrid_clock.cc:648] HybridClock initialized: now 1790311480163496 us; error 35 us; skew 500 ppm
Sep 25 04:44:40 dist-test-slave-kghz krb5kdc[5947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311480, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:44:40.169927 5960 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260925 04:44:40.170939 5960 webserver.cc:460] Webserver started at http://127.0.98.126:37095/ using document root <none> and password file <none>
I20260925 04:44:40.171499 5960 fs_manager.cc:359] Metadata directory not provided
I20260925 04:44:40.171561 5960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:44:40.171725 5960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:44:40.173767 5960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/minicluster-data/master-0/data/instance:
uuid: "4279b8711f0e40b28afb74f212cdc0ef"
format_stamp: "Formatted at 2026-09-25 04:44:40 on dist-test-slave-kghz"
server_key: "4323771a53927ce6c8ca2edff3bc1b55"
server_key_iv: "e41e405d3be34400169984b07b5278f0"
server_key_version: "encryptionkey@0"
I20260925 04:44:40.174538 5960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/minicluster-data/master-0/wal/instance:
uuid: "4279b8711f0e40b28afb74f212cdc0ef"
format_stamp: "Formatted at 2026-09-25 04:44:40 on dist-test-slave-kghz"
server_key: "4323771a53927ce6c8ca2edff3bc1b55"
server_key_iv: "e41e405d3be34400169984b07b5278f0"
server_key_version: "encryptionkey@0"
I20260925 04:44:40.178866 5960 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 04:44:40.181396 5975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:40.182740 5960 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:44:40.182889 5960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/minicluster-data/master-0/wal
uuid: "4279b8711f0e40b28afb74f212cdc0ef"
format_stamp: "Formatted at 2026-09-25 04:44:40 on dist-test-slave-kghz"
server_key: "4323771a53927ce6c8ca2edff3bc1b55"
server_key_iv: "e41e405d3be34400169984b07b5278f0"
server_key_version: "encryptionkey@0"
I20260925 04:44:40.182992 5960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:44:40.197091 5960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:44:40.200605 5960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:44:40.200819 5960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:44:40.207971 5960 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:34095
I20260925 04:44:40.208292 6027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:34095 every 8 connection(s)
I20260925 04:44:40.209053 5960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
I20260925 04:44:40.212042 6028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:44:40.213373 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 5960
I20260925 04:44:40.213508 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/minicluster-data/master-0/wal/instance
I20260925 04:44:40.213797 393 external_mini_cluster.cc:1428] Setting key 69095d3079b856cce2e004f5d996317f
I20260925 04:44:40.218314 6028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4279b8711f0e40b28afb74f212cdc0ef: Bootstrap starting.
I20260925 04:44:40.220556 6028 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4279b8711f0e40b28afb74f212cdc0ef: Neither blocks nor log segments found. Creating new log.
I20260925 04:44:40.221292 6028 log.cc:824] T 00000000000000000000000000000000 P 4279b8711f0e40b28afb74f212cdc0ef: Log is configured to *not* fsync() on all Append() calls
Sep 25 04:44:40 dist-test-slave-kghz krb5kdc[5947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:44:40.223392 6028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4279b8711f0e40b28afb74f212cdc0ef: No bootstrap required, opened a new log
I20260925 04:44:40.227293 6028 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4279b8711f0e40b28afb74f212cdc0ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4279b8711f0e40b28afb74f212cdc0ef" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34095 } }
I20260925 04:44:40.227568 6028 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4279b8711f0e40b28afb74f212cdc0ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:44:40.227660 6028 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4279b8711f0e40b28afb74f212cdc0ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4279b8711f0e40b28afb74f212cdc0ef, State: Initialized, Role: FOLLOWER
I20260925 04:44:40.228158 6031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:40.215330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50328 (local address 127.0.98.126:34095)
0925 04:44:40.215786 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:40.215801 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:40.215836 (+ 35us) server_negotiation.cc:407] Connection header received
0925 04:44:40.216483 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:40.216502 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:40.216838 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:40.217199 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:40.218009 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:40.219089 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:40.219931 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:40.220412 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:40.220596 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:40.223313 (+ 2717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:40.223318 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:40.223332 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:40.223358 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:40.225183 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:40.225635 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:40.225647 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:40.225653 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:40.225760 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:40.226054 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:40.226058 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:40.226060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:40.226526 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:40.226737 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:40.227005 (+ 268us) server_negotiation.cc:299] Negotiation successful
0925 04:44:40.227299 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":141,"threads_started":1}
I20260925 04:44:40.228163 6028 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4279b8711f0e40b28afb74f212cdc0ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4279b8711f0e40b28afb74f212cdc0ef" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34095 } }
I20260925 04:44:40.228405 6028 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4279b8711f0e40b28afb74f212cdc0ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:44:40.228484 6028 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4279b8711f0e40b28afb74f212cdc0ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:44:40.228585 6028 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4279b8711f0e40b28afb74f212cdc0ef [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:44:40.229509 6028 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4279b8711f0e40b28afb74f212cdc0ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4279b8711f0e40b28afb74f212cdc0ef" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34095 } }
I20260925 04:44:40.229861 6028 leader_election.cc:302] T 00000000000000000000000000000000 P 4279b8711f0e40b28afb74f212cdc0ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4279b8711f0e40b28afb74f212cdc0ef; no voters:
I20260925 04:44:40.230187 6028 leader_election.cc:288] T 00000000000000000000000000000000 P 4279b8711f0e40b28afb74f212cdc0ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:44:40.230335 6033 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4279b8711f0e40b28afb74f212cdc0ef [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:44:40.230620 6033 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4279b8711f0e40b28afb74f212cdc0ef [term 1 LEADER]: Becoming Leader. State: Replica: 4279b8711f0e40b28afb74f212cdc0ef, State: Running, Role: LEADER
I20260925 04:44:40.230947 6033 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4279b8711f0e40b28afb74f212cdc0ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4279b8711f0e40b28afb74f212cdc0ef" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34095 } }
I20260925 04:44:40.231530 6028 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4279b8711f0e40b28afb74f212cdc0ef [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:44:40.232126 6033 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4279b8711f0e40b28afb74f212cdc0ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4279b8711f0e40b28afb74f212cdc0ef. Latest consensus state: current_term: 1 leader_uuid: "4279b8711f0e40b28afb74f212cdc0ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4279b8711f0e40b28afb74f212cdc0ef" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34095 } } }
I20260925 04:44:40.232282 6033 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4279b8711f0e40b28afb74f212cdc0ef [sys.catalog]: This master's current role is: LEADER
I20260925 04:44:40.232643 6035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4279b8711f0e40b28afb74f212cdc0ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4279b8711f0e40b28afb74f212cdc0ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4279b8711f0e40b28afb74f212cdc0ef" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34095 } } }
I20260925 04:44:40.232717 6039 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:44:40.232800 6035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4279b8711f0e40b28afb74f212cdc0ef [sys.catalog]: This master's current role is: LEADER
I20260925 04:44:40.235837 6039 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:44:40.242588 6039 catalog_manager.cc:1347] Generated new cluster ID: 6ed7351701fb475681e415062c9bbd6b
I20260925 04:44:40.242703 6039 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:44:40.289440 6039 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:44:40.290791 6039 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:44:40.300578 6039 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4279b8711f0e40b28afb74f212cdc0ef: Generated new TSK 0
I20260925 04:44:40.301527 6039 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/krb5kdc/kudu_127.0.98.65.keytab.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/krb5kdc/kudu_127.0.98.65.keytab.
I20260925 04:44:40.342633 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:0
--local_ip_for_outbound_sockets=127.0.98.65
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:34095
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:40049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:44:40.456506 6054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:44:40.456787 6054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:44:40.456833 6054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:44:40.461030 6054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:44:40.461097 6054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:44:40.461195 6054 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260925 04:44:40.466542 6054 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:40049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:34095
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6054
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:44:40.467628 6054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:44:40.468653 6054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:44:40.478365 6059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:40.478379 6060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:40.478461 6054 server_base.cc:1034] running on GCE node
W20260925 04:44:40.478673 6062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:40.479067 6054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:44:40.479717 6054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:44:40.480880 6054 hybrid_clock.cc:648] HybridClock initialized: now 1790311480480861 us; error 32 us; skew 500 ppm
Sep 25 04:44:40 dist-test-slave-kghz krb5kdc[5947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311480, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:44:40.483862 6054 init.cc:377] Logged in from keytab as kudu/127.0.98.65@KRBTEST.COM (short username kudu)
I20260925 04:44:40.484764 6054 webserver.cc:460] Webserver started at http://127.0.98.65:33217/ using document root <none> and password file <none>
I20260925 04:44:40.485327 6054 fs_manager.cc:359] Metadata directory not provided
I20260925 04:44:40.485387 6054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:44:40.485615 6054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:44:40.487841 6054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/minicluster-data/ts-0/data/instance:
uuid: "cb01a967a24546cfb06bfc6841cd5b21"
format_stamp: "Formatted at 2026-09-25 04:44:40 on dist-test-slave-kghz"
server_key: "45e18afc2a8447a472a4fec400971b38"
server_key_iv: "8ab2c5aa99b5d440298a7ea57f4849e9"
server_key_version: "encryptionkey@0"
I20260925 04:44:40.488400 6054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/minicluster-data/ts-0/wal/instance:
uuid: "cb01a967a24546cfb06bfc6841cd5b21"
format_stamp: "Formatted at 2026-09-25 04:44:40 on dist-test-slave-kghz"
server_key: "45e18afc2a8447a472a4fec400971b38"
server_key_iv: "8ab2c5aa99b5d440298a7ea57f4849e9"
server_key_version: "encryptionkey@0"
I20260925 04:44:40.492130 6054 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 04:44:40.494598 6069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:40.495959 6054 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 04:44:40.496165 6054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/minicluster-data/ts-0/wal
uuid: "cb01a967a24546cfb06bfc6841cd5b21"
format_stamp: "Formatted at 2026-09-25 04:44:40 on dist-test-slave-kghz"
server_key: "45e18afc2a8447a472a4fec400971b38"
server_key_iv: "8ab2c5aa99b5d440298a7ea57f4849e9"
server_key_version: "encryptionkey@0"
I20260925 04:44:40.496292 6054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:44:40.528646 6054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:44:40.532217 6054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:44:40.532456 6054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:44:40.533147 6054 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:44:40.534022 6054 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:44:40.534092 6054 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:40.534164 6054 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:44:40.534209 6054 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:40.543265 6054 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:45449
I20260925 04:44:40.543311 6182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:45449 every 8 connection(s)
I20260925 04:44:40.544234 6054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
I20260925 04:44:40.549721 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 6054
I20260925 04:44:40.549860 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/minicluster-data/ts-0/wal/instance
I20260925 04:44:40.550205 393 external_mini_cluster.cc:1428] Setting key 6fcba0d600ae6d8e588ed4ee2abd3112
Sep 25 04:44:40 dist-test-slave-kghz krb5kdc[5947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311480, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:44:40.559505 6031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:40.546147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:37889 (local address 127.0.98.126:34095)
0925 04:44:40.546372 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:40.546376 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:40.547226 (+ 850us) server_negotiation.cc:407] Connection header received
0925 04:44:40.548581 (+ 1355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:40.548583 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:40.548629 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:40.548708 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:40.550120 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:40.550806 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:40.552322 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:40.552553 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:40.552647 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:40.555512 (+ 2865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:40.555515 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:40.555517 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:40.555540 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:40.557185 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:40.557839 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:40.557841 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:40.557842 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:40.557916 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:40.558327 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:40.558329 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:40.558330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:40.558514 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:40.558592 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:40.559276 (+ 684us) server_negotiation.cc:299] Negotiation successful
0925 04:44:40.559382 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
WARNING: no policy specified for kudu/127.0.98.66@KRBTEST.COM; defaulting to no policy
I20260925 04:44:40.560750 6185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:40.546572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34095 (local address 127.0.98.65:37889)
0925 04:44:40.547048 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:40.547111 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:40.548376 (+ 1265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:40.548873 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:40.548880 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:40.549270 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:40.549950 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:40.549972 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:40.550923 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:40.550927 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:40.552177 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:40.552192 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:40.552712 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:40.552718 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:40.552973 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:40.553769 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:40.553785 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:40.555330 (+ 1545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:44:40.557310 (+ 1980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:40.557318 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:40.557335 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:40.557730 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:40.558008 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:40.558014 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:40.558018 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:40.558212 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:40.558627 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:44:40.558634 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:44:40.559002 (+ 368us) client_negotiation.cc:769] Sending connection context
0925 04:44:40.559224 (+ 222us) client_negotiation.cc:240] Negotiation successful
0925 04:44:40.559608 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":96,"threads_started":1}
I20260925 04:44:40.562062 6183 heartbeater.cc:347] Connected to a master server at 127.0.98.126:34095
I20260925 04:44:40.562377 6183 heartbeater.cc:464] Registering TS with master...
I20260925 04:44:40.562852 6183 heartbeater.cc:511] Master 127.0.98.126:34095 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.66@KRBTEST.COM" created.
I20260925 04:44:40.564522 5992 ts_manager.cc:194] Registered new tserver with Master: cb01a967a24546cfb06bfc6841cd5b21 (127.0.98.65:45449)
I20260925 04:44:40.565670 5992 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.65@KRBTEST.COM'} at 127.0.98.65:37889
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/krb5kdc/kudu_127.0.98.66.keytab.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/krb5kdc/kudu_127.0.98.66.keytab.
I20260925 04:44:40.579850 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:0
--local_ip_for_outbound_sockets=127.0.98.66
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:34095
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:40049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:44:40.695385 6188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:44:40.695655 6188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:44:40.695724 6188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:44:40.699900 6188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:44:40.700019 6188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:44:40.700119 6188 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260925 04:44:40.705523 6188 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:40049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:34095
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6188
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:44:40.706637 6188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:44:40.707669 6188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:44:40.717494 6196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:40.717475 6194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:40.717481 6193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:40.717558 6188 server_base.cc:1034] running on GCE node
I20260925 04:44:40.718096 6188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:44:40.718775 6188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:44:40.719938 6188 hybrid_clock.cc:648] HybridClock initialized: now 1790311480719923 us; error 28 us; skew 500 ppm
Sep 25 04:44:40 dist-test-slave-kghz krb5kdc[5947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311480, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:44:40.722890 6188 init.cc:377] Logged in from keytab as kudu/127.0.98.66@KRBTEST.COM (short username kudu)
I20260925 04:44:40.723786 6188 webserver.cc:460] Webserver started at http://127.0.98.66:38031/ using document root <none> and password file <none>
I20260925 04:44:40.724351 6188 fs_manager.cc:359] Metadata directory not provided
I20260925 04:44:40.724412 6188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:44:40.724650 6188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:44:40.726765 6188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/minicluster-data/ts-1/data/instance:
uuid: "3b4f55cc716b4bcfa9bba7d6aedf0077"
format_stamp: "Formatted at 2026-09-25 04:44:40 on dist-test-slave-kghz"
server_key: "5d0bfb04920da4e09c6f42bcc72dfbaf"
server_key_iv: "bed6e3f5571bd7ca86f1ab51ab0bb40b"
server_key_version: "encryptionkey@0"
I20260925 04:44:40.727319 6188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/minicluster-data/ts-1/wal/instance:
uuid: "3b4f55cc716b4bcfa9bba7d6aedf0077"
format_stamp: "Formatted at 2026-09-25 04:44:40 on dist-test-slave-kghz"
server_key: "5d0bfb04920da4e09c6f42bcc72dfbaf"
server_key_iv: "bed6e3f5571bd7ca86f1ab51ab0bb40b"
server_key_version: "encryptionkey@0"
I20260925 04:44:40.731073 6188 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 04:44:40.733762 6203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:40.735047 6188 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 04:44:40.735183 6188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/minicluster-data/ts-1/wal
uuid: "3b4f55cc716b4bcfa9bba7d6aedf0077"
format_stamp: "Formatted at 2026-09-25 04:44:40 on dist-test-slave-kghz"
server_key: "5d0bfb04920da4e09c6f42bcc72dfbaf"
server_key_iv: "bed6e3f5571bd7ca86f1ab51ab0bb40b"
server_key_version: "encryptionkey@0"
I20260925 04:44:40.735291 6188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:44:40.758371 6188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:44:40.761606 6188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:44:40.761830 6188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:44:40.762553 6188 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:44:40.763419 6188 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:44:40.763471 6188 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:40.763513 6188 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:44:40.763564 6188 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:40.772670 6188 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:45501
I20260925 04:44:40.772689 6316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:45501 every 8 connection(s)
I20260925 04:44:40.773581 6188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
I20260925 04:44:40.777143 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 6188
I20260925 04:44:40.777277 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/minicluster-data/ts-1/wal/instance
I20260925 04:44:40.777632 393 external_mini_cluster.cc:1428] Setting key 7721d12eb8278ecab6456896ed07d185
Sep 25 04:44:40 dist-test-slave-kghz krb5kdc[5947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311480, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.67@KRBTEST.COM; defaulting to no policy
I20260925 04:44:40.789628 6031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:40.775421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:52089 (local address 127.0.98.126:34095)
0925 04:44:40.775562 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:40.775566 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:40.776434 (+ 868us) server_negotiation.cc:407] Connection header received
0925 04:44:40.778350 (+ 1916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:40.778354 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:40.778433 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:40.778534 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:40.780146 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:40.780726 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:40.782169 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:40.782484 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:40.782599 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:40.785485 (+ 2886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:40.785488 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:40.785491 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:40.785515 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:40.787354 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:40.788042 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:40.788045 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:40.788046 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:40.788128 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:40.788515 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:40.788518 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:40.788519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:40.788709 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:40.788781 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:40.789378 (+ 597us) server_negotiation.cc:299] Negotiation successful
0925 04:44:40.789500 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":16,"server-negotiator.queue_time_us":56}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.67@KRBTEST.COM" created.
I20260925 04:44:40.790783 6319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:40.775819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34095 (local address 127.0.98.66:52089)
0925 04:44:40.776266 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:40.776308 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:40.778057 (+ 1749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:40.778685 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:40.778692 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:40.779242 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:40.779976 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:40.780006 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:40.780852 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:40.780856 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:40.782011 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:40.782033 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:40.782635 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:40.782639 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:40.782819 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:40.783437 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:40.783453 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:40.785295 (+ 1842us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:44:40.787527 (+ 2232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:40.787536 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:40.787555 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:40.787934 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:40.788225 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:40.788227 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:40.788228 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:40.788416 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:40.788809 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:44:40.788816 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:44:40.789114 (+ 298us) client_negotiation.cc:769] Sending connection context
0925 04:44:40.789313 (+ 199us) client_negotiation.cc:240] Negotiation successful
0925 04:44:40.789709 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":100,"threads_started":1}
I20260925 04:44:40.792057 6317 heartbeater.cc:347] Connected to a master server at 127.0.98.126:34095
I20260925 04:44:40.792317 6317 heartbeater.cc:464] Registering TS with master...
I20260925 04:44:40.792826 6317 heartbeater.cc:511] Master 127.0.98.126:34095 requested a full tablet report, sending...
I20260925 04:44:40.793921 5992 ts_manager.cc:194] Registered new tserver with Master: 3b4f55cc716b4bcfa9bba7d6aedf0077 (127.0.98.66:45501)
I20260925 04:44:40.794574 5992 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.66@KRBTEST.COM'} at 127.0.98.66:52089
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/krb5kdc/kudu_127.0.98.67.keytab.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/krb5kdc/kudu_127.0.98.67.keytab.
I20260925 04:44:40.805714 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:0
--local_ip_for_outbound_sockets=127.0.98.67
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:34095
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:40049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:44:40.918335 6322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:44:40.918596 6322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:44:40.918639 6322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:44:40.922722 6322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:44:40.922780 6322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:44:40.922839 6322 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260925 04:44:40.928094 6322 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:40049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:34095
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6322
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:44:40.929113 6322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:44:40.930076 6322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 04:44:40.940363 6322 server_base.cc:1034] running on GCE node
W20260925 04:44:40.940362 6328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:40.940588 6327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:40.940372 6330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:40.941041 6322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:44:40.941700 6322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:44:40.942886 6322 hybrid_clock.cc:648] HybridClock initialized: now 1790311480942867 us; error 30 us; skew 500 ppm
Sep 25 04:44:40 dist-test-slave-kghz krb5kdc[5947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311480, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:44:40.945713 6322 init.cc:377] Logged in from keytab as kudu/127.0.98.67@KRBTEST.COM (short username kudu)
I20260925 04:44:40.946738 6322 webserver.cc:460] Webserver started at http://127.0.98.67:46651/ using document root <none> and password file <none>
I20260925 04:44:40.947314 6322 fs_manager.cc:359] Metadata directory not provided
I20260925 04:44:40.947403 6322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:44:40.947638 6322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:44:40.949751 6322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/minicluster-data/ts-2/data/instance:
uuid: "b4edb0af01094261bb5f5fa72da45981"
format_stamp: "Formatted at 2026-09-25 04:44:40 on dist-test-slave-kghz"
server_key: "cb1f9a69498ee04e82ba53f2de236771"
server_key_iv: "1d5dcd16f273836bae055556d56e2e76"
server_key_version: "encryptionkey@0"
I20260925 04:44:40.950412 6322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/minicluster-data/ts-2/wal/instance:
uuid: "b4edb0af01094261bb5f5fa72da45981"
format_stamp: "Formatted at 2026-09-25 04:44:40 on dist-test-slave-kghz"
server_key: "cb1f9a69498ee04e82ba53f2de236771"
server_key_iv: "1d5dcd16f273836bae055556d56e2e76"
server_key_version: "encryptionkey@0"
I20260925 04:44:40.954455 6322 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 04:44:40.957051 6337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:40.958474 6322 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:44:40.958616 6322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/minicluster-data/ts-2/wal
uuid: "b4edb0af01094261bb5f5fa72da45981"
format_stamp: "Formatted at 2026-09-25 04:44:40 on dist-test-slave-kghz"
server_key: "cb1f9a69498ee04e82ba53f2de236771"
server_key_iv: "1d5dcd16f273836bae055556d56e2e76"
server_key_version: "encryptionkey@0"
I20260925 04:44:40.958737 6322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:44:40.969987 6322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:44:40.973378 6322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:44:40.973588 6322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:44:40.974171 6322 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:44:40.975032 6322 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:44:40.975102 6322 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:40.975175 6322 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:44:40.975217 6322 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:40.984480 6322 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:38597
I20260925 04:44:40.984541 6450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:38597 every 8 connection(s)
I20260925 04:44:40.985388 6322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
I20260925 04:44:40.992300 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 6322
I20260925 04:44:40.992449 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/minicluster-data/ts-2/wal/instance
I20260925 04:44:40.992744 393 external_mini_cluster.cc:1428] Setting key e135b04363a4ca64a89079d8f4094d5b
Sep 25 04:44:40 dist-test-slave-kghz krb5kdc[5947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311480, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:44:40.999574 6031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:40.987288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:60985 (local address 127.0.98.126:34095)
0925 04:44:40.987472 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:40.987476 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:40.988354 (+ 878us) server_negotiation.cc:407] Connection header received
0925 04:44:40.989691 (+ 1337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:40.989700 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:40.989746 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:40.989830 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:40.991011 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:40.991525 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:40.993163 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:40.993396 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:40.993520 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:40.996271 (+ 2751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:40.996274 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:40.996277 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:40.996301 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:40.997561 (+ 1260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:40.998099 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:40.998102 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:40.998103 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:40.998173 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:40.998576 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:40.998578 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:40.998579 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:40.998747 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:40.998832 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:40.999348 (+ 516us) server_negotiation.cc:299] Negotiation successful
0925 04:44:40.999449 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 04:44:41.000512 6453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:40.987745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34095 (local address 127.0.98.67:60985)
0925 04:44:40.988193 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:40.988236 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:40.989494 (+ 1258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:40.989985 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:40.989992 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:40.990363 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:40.990850 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:40.990864 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:40.991663 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:40.991666 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:40.992995 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:40.993006 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:40.993557 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:40.993561 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:40.993762 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:40.994482 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:40.994499 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:40.996117 (+ 1618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:44:40.997697 (+ 1580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:40.997702 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:40.997714 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:40.998006 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:40.998329 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:40.998333 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:40.998334 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:40.998489 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:40.998873 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:44:40.998878 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:44:40.999124 (+ 246us) client_negotiation.cc:769] Sending connection context
0925 04:44:40.999281 (+ 157us) client_negotiation.cc:240] Negotiation successful
0925 04:44:40.999547 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":101,"threads_started":1}
I20260925 04:44:41.001753 6451 heartbeater.cc:347] Connected to a master server at 127.0.98.126:34095
I20260925 04:44:41.001981 6451 heartbeater.cc:464] Registering TS with master...
I20260925 04:44:41.002583 6451 heartbeater.cc:511] Master 127.0.98.126:34095 requested a full tablet report, sending...
I20260925 04:44:41.003582 5992 ts_manager.cc:194] Registered new tserver with Master: b4edb0af01094261bb5f5fa72da45981 (127.0.98.67:38597)
I20260925 04:44:41.004150 5992 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.67@KRBTEST.COM'} at 127.0.98.67:60985
I20260925 04:44:41.006865 393 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/krb5kdc/test-admin.keytab.
Sep 25 04:44:41 dist-test-slave-kghz krb5kdc[5947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:44:41.023205 393 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260925 04:44:41.026364 6458 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:44:41.024677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35456 (local address 127.0.98.65:45449)
0925 04:44:41.025045 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:41.025061 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:41.025085 (+ 24us) server_negotiation.cc:407] Connection header received
0925 04:44:41.025136 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:41.025141 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:41.025332 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:41.025469 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:41.026131 (+ 662us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35456: BlockingRecv error: recv got EOF from 127.0.0.1:35456 (error 108)
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":105,"threads_started":1}
W20260925 04:44:41.026368 393 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/krb5kdc/krb5cc' of type FILE
Sep 25 04:44:41 dist-test-slave-kghz krb5kdc[5947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:44:41.027609 393 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 04:44:41 dist-test-slave-kghz krb5kdc[5947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:44:41.028507 393 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 04:44:41 dist-test-slave-kghz krb5kdc[5947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.65@KRBTEST.COM
I20260925 04:44:41.040009 6458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:41.029423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35470 (local address 127.0.98.65:45449)
0925 04:44:41.029561 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:41.029564 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:41.029580 (+ 16us) server_negotiation.cc:407] Connection header received
0925 04:44:41.029731 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:41.029734 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:41.029780 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:41.029859 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:41.030455 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:41.031251 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:41.032272 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:41.032709 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:41.032797 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:41.035295 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:41.035304 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:41.035318 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:41.035342 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:41.037528 (+ 2186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:41.038124 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:41.038131 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:41.038133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:41.038232 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:41.038623 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:41.038645 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:41.038649 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:41.039003 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:41.039266 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:41.039541 (+ 275us) server_negotiation.cc:299] Negotiation successful
0925 04:44:41.039771 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260925 04:44:41.044682 6458 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:44:41.043684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35484 (local address 127.0.98.65:45449)
0925 04:44:41.043886 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:41.043890 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:41.043903 (+ 13us) server_negotiation.cc:407] Connection header received
0925 04:44:41.044010 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:41.044013 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:41.044052 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:41.044143 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:41.044578 (+ 435us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35484: BlockingRecv error: recv got EOF from 127.0.0.1:35484 (error 108)
Metrics: {"server-negotiator.queue_time_us":114}
W20260925 04:44:41.045341 393 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/krb5kdc/krb5cc)
Sep 25 04:44:41 dist-test-slave-kghz krb5kdc[5947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:44:41.046535 393 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 04:44:41 dist-test-slave-kghz krb5kdc[5947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:44:41.047411 393 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 04:44:41 dist-test-slave-kghz krb5kdc[5947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.65@KRBTEST.COM
I20260925 04:44:41.056828 6458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:41.048211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35494 (local address 127.0.98.65:45449)
0925 04:44:41.048337 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:41.048341 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:41.048466 (+ 125us) server_negotiation.cc:407] Connection header received
0925 04:44:41.048606 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:41.048609 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:41.048648 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:41.048733 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:41.049228 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:41.049757 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:41.050640 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:41.050886 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:41.051005 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:41.053721 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:41.053727 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:41.053735 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:41.053757 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:41.055227 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:41.055662 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:41.055666 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:41.055668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:41.055748 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:41.056023 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:41.056028 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:41.056030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:41.056193 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:41.056328 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:41.056564 (+ 236us) server_negotiation.cc:299] Negotiation successful
0925 04:44:41.056670 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260925 04:44:41.059587 6458 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:44:41.058641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35508 (local address 127.0.98.65:45449)
0925 04:44:41.058758 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:41.058762 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:41.058872 (+ 110us) server_negotiation.cc:407] Connection header received
0925 04:44:41.058956 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:41.058958 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:41.058993 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:41.059051 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:41.059496 (+ 445us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35508: BlockingRecv error: recv got EOF from 127.0.0.1:35508 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260925 04:44:41.059705 393 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790311449873301-393-0/krb5kdc/krb5cc' of type FILE
Sep 25 04:44:41 dist-test-slave-kghz krb5kdc[5947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:44:41.060712 393 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 04:44:41 dist-test-slave-kghz krb5kdc[5947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:44:41.061714 393 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 04:44:41 dist-test-slave-kghz krb5kdc[5947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.65@KRBTEST.COM
I20260925 04:44:41.070595 6458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:41.062638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35524 (local address 127.0.98.65:45449)
0925 04:44:41.062744 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:41.062748 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:41.063007 (+ 259us) server_negotiation.cc:407] Connection header received
0925 04:44:41.063046 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:41.063048 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:41.063087 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:41.063157 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:41.063652 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:41.064142 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:41.064909 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:41.065159 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:41.065248 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:41.067454 (+ 2206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:41.067457 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:41.067459 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:41.067481 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:41.068953 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:41.069340 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:41.069342 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:41.069343 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:41.069420 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:41.069676 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:41.069679 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:41.069681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:41.069868 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:41.069978 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:41.070234 (+ 256us) server_negotiation.cc:299] Negotiation successful
0925 04:44:41.070389 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 04:44:41.071648 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 6054
I20260925 04:44:41.078397 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 6188
I20260925 04:44:41.083648 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 6322
I20260925 04:44:41.088994 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 5960
2026-09-25T04:44:41Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2767 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:44:41 dist-test-slave-kghz krb5kdc[6471](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:44:41 dist-test-slave-kghz krb5kdc[6471](info): set up 2 sockets
Sep 25 04:44:41 dist-test-slave-kghz krb5kdc[6471](info): commencing operation
krb5kdc: starting...
W20260925 04:44:42.134801 393 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/krb5kdc/test-user.keytab.
Sep 25 04:44:42 dist-test-slave-kghz krb5kdc[6471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:44:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:44:42Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/krb5kdc/oryx_127.0.98.126.keytab.
Entry for principal oryx/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/krb5kdc/oryx_127.0.98.126.keytab.
I20260925 04:44:42.261430 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:46181
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/krb5kdc/oryx_127.0.98.126.keytab
--principal=oryx/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:33935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:46181
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:44:42.373464 6484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:44:42.373795 6484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:44:42.373878 6484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:44:42.377938 6484 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:44:42.378041 6484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:44:42.378082 6484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:44:42.378141 6484 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:44:42.378181 6484 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:44:42.383652 6484 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:33935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:46181
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/krb5kdc/oryx_127.0.98.126.keytab
--principal=oryx/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:46181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6484
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:44:42.384752 6484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:44:42.385778 6484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 04:44:42.394644 6484 server_base.cc:1034] running on GCE node
W20260925 04:44:42.394613 6492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:42.394732 6490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:42.394625 6489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:42.395399 6484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:44:42.396559 6484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:44:42.397742 6484 hybrid_clock.cc:648] HybridClock initialized: now 1790311482397720 us; error 36 us; skew 500 ppm
Sep 25 04:44:42 dist-test-slave-kghz krb5kdc[6471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311482, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:44:42.400457 6484 init.cc:377] Logged in from keytab as oryx/127.0.98.126@KRBTEST.COM (short username oryx)
I20260925 04:44:42.401329 6484 webserver.cc:460] Webserver started at http://127.0.98.126:35723/ using document root <none> and password file <none>
I20260925 04:44:42.401845 6484 fs_manager.cc:359] Metadata directory not provided
I20260925 04:44:42.401901 6484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:44:42.402063 6484 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:44:42.404117 6484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/master-0/data/instance:
uuid: "58edb7db5329430685f0d9e7a5098394"
format_stamp: "Formatted at 2026-09-25 04:44:42 on dist-test-slave-kghz"
server_key: "6fc1d33a3666c86bac175d2362b4e5e9"
server_key_iv: "0ecd399c1e4893453b667ca9280f4117"
server_key_version: "encryptionkey@0"
I20260925 04:44:42.404644 6484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/master-0/wal/instance:
uuid: "58edb7db5329430685f0d9e7a5098394"
format_stamp: "Formatted at 2026-09-25 04:44:42 on dist-test-slave-kghz"
server_key: "6fc1d33a3666c86bac175d2362b4e5e9"
server_key_iv: "0ecd399c1e4893453b667ca9280f4117"
server_key_version: "encryptionkey@0"
I20260925 04:44:42.408429 6484 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 04:44:42.411027 6499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:42.412401 6484 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:44:42.412551 6484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/master-0/wal
uuid: "58edb7db5329430685f0d9e7a5098394"
format_stamp: "Formatted at 2026-09-25 04:44:42 on dist-test-slave-kghz"
server_key: "6fc1d33a3666c86bac175d2362b4e5e9"
server_key_iv: "0ecd399c1e4893453b667ca9280f4117"
server_key_version: "encryptionkey@0"
I20260925 04:44:42.412693 6484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:44:42.428335 6484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:44:42.431803 6484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:44:42.432086 6484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:44:42.439781 6484 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:46181
I20260925 04:44:42.440096 6551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:46181 every 8 connection(s)
I20260925 04:44:42.440807 6484 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
I20260925 04:44:42.444057 6552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:44:42.448069 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 6484
I20260925 04:44:42.448208 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/master-0/wal/instance
I20260925 04:44:42.448504 393 external_mini_cluster.cc:1428] Setting key 45ebf9101c4ce241863d7709489ecfc3
I20260925 04:44:42.450196 6552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58edb7db5329430685f0d9e7a5098394: Bootstrap starting.
I20260925 04:44:42.452445 6552 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58edb7db5329430685f0d9e7a5098394: Neither blocks nor log segments found. Creating new log.
I20260925 04:44:42.453183 6552 log.cc:824] T 00000000000000000000000000000000 P 58edb7db5329430685f0d9e7a5098394: Log is configured to *not* fsync() on all Append() calls
I20260925 04:44:42.455101 6552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58edb7db5329430685f0d9e7a5098394: No bootstrap required, opened a new log
Sep 25 04:44:42 dist-test-slave-kghz krb5kdc[6471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM
I20260925 04:44:42.458688 6552 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58edb7db5329430685f0d9e7a5098394 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58edb7db5329430685f0d9e7a5098394" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46181 } }
I20260925 04:44:42.458938 6552 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58edb7db5329430685f0d9e7a5098394 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:44:42.459029 6552 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58edb7db5329430685f0d9e7a5098394 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58edb7db5329430685f0d9e7a5098394, State: Initialized, Role: FOLLOWER
I20260925 04:44:42.459578 6552 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58edb7db5329430685f0d9e7a5098394 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58edb7db5329430685f0d9e7a5098394" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46181 } }
I20260925 04:44:42.459738 6552 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58edb7db5329430685f0d9e7a5098394 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:44:42.459802 6552 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58edb7db5329430685f0d9e7a5098394 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:44:42.459899 6552 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58edb7db5329430685f0d9e7a5098394 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:44:42.461012 6552 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58edb7db5329430685f0d9e7a5098394 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58edb7db5329430685f0d9e7a5098394" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46181 } }
I20260925 04:44:42.461400 6552 leader_election.cc:302] T 00000000000000000000000000000000 P 58edb7db5329430685f0d9e7a5098394 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58edb7db5329430685f0d9e7a5098394; no voters:
I20260925 04:44:42.461753 6552 leader_election.cc:288] T 00000000000000000000000000000000 P 58edb7db5329430685f0d9e7a5098394 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:44:42.461946 6557 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58edb7db5329430685f0d9e7a5098394 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:44:42.462304 6557 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58edb7db5329430685f0d9e7a5098394 [term 1 LEADER]: Becoming Leader. State: Replica: 58edb7db5329430685f0d9e7a5098394, State: Running, Role: LEADER
I20260925 04:44:42.462695 6557 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58edb7db5329430685f0d9e7a5098394 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58edb7db5329430685f0d9e7a5098394" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46181 } }
I20260925 04:44:42.463076 6552 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58edb7db5329430685f0d9e7a5098394 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:44:42.463975 6555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:42.449991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39334 (local address 127.0.98.126:46181)
0925 04:44:42.450507 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:42.450524 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:42.450559 (+ 35us) server_negotiation.cc:407] Connection header received
0925 04:44:42.451191 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:42.451208 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:42.451567 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:42.451933 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:42.452776 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:42.453834 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:42.454758 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:42.455301 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:42.455485 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:42.457978 (+ 2493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:42.457984 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:42.457997 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:42.458024 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:42.460145 (+ 2121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:42.460609 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:42.460615 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:42.460621 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:42.460769 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:42.461125 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:42.461128 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:42.461130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:42.461659 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:42.461857 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:42.462405 (+ 548us) server_negotiation.cc:299] Negotiation successful
0925 04:44:42.462736 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":129,"threads_started":1}
I20260925 04:44:42.464391 6558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58edb7db5329430685f0d9e7a5098394 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "58edb7db5329430685f0d9e7a5098394" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58edb7db5329430685f0d9e7a5098394" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46181 } } }
I20260925 04:44:42.464521 6558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58edb7db5329430685f0d9e7a5098394 [sys.catalog]: This master's current role is: LEADER
I20260925 04:44:42.464797 6559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58edb7db5329430685f0d9e7a5098394 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58edb7db5329430685f0d9e7a5098394. Latest consensus state: current_term: 1 leader_uuid: "58edb7db5329430685f0d9e7a5098394" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58edb7db5329430685f0d9e7a5098394" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46181 } } }
I20260925 04:44:42.464871 6559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58edb7db5329430685f0d9e7a5098394 [sys.catalog]: This master's current role is: LEADER
I20260925 04:44:42.465114 6567 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:44:42.468257 6567 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:44:42.474462 6567 catalog_manager.cc:1347] Generated new cluster ID: 279c26e79db84d6982461caf8787f885
I20260925 04:44:42.474555 6567 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:44:42.487004 6567 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:44:42.487987 6567 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:44:42.500835 6567 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 58edb7db5329430685f0d9e7a5098394: Generated new TSK 0
I20260925 04:44:42.501613 6567 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/krb5kdc/oryx_127.0.98.65.keytab.
Entry for principal oryx/127.0.98.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/krb5kdc/oryx_127.0.98.65.keytab.
I20260925 04:44:42.530992 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:0
--local_ip_for_outbound_sockets=127.0.98.65
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:46181
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/krb5kdc/oryx_127.0.98.65.keytab
--principal=oryx/127.0.98.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:33935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:44:42.660953 6578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:44:42.661288 6578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:44:42.661334 6578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:44:42.665752 6578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:44:42.665815 6578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:44:42.665917 6578 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260925 04:44:42.671319 6578 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:33935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/krb5kdc/oryx_127.0.98.65.keytab
--principal=oryx/127.0.98.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:46181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:44:42.672369 6578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:44:42.673404 6578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:44:42.683700 6584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:42.683706 6586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:42.683706 6583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:42.684188 6578 server_base.cc:1034] running on GCE node
I20260925 04:44:42.684576 6578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:44:42.685238 6578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:44:42.686416 6578 hybrid_clock.cc:648] HybridClock initialized: now 1790311482686404 us; error 31 us; skew 500 ppm
Sep 25 04:44:42 dist-test-slave-kghz krb5kdc[6471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311482, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:44:42.689320 6578 init.cc:377] Logged in from keytab as oryx/127.0.98.65@KRBTEST.COM (short username oryx)
I20260925 04:44:42.690330 6578 webserver.cc:460] Webserver started at http://127.0.98.65:46493/ using document root <none> and password file <none>
I20260925 04:44:42.690927 6578 fs_manager.cc:359] Metadata directory not provided
I20260925 04:44:42.691025 6578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:44:42.691269 6578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:44:42.693370 6578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-0/data/instance:
uuid: "0bbea834c4b244998ef191efd5bb4173"
format_stamp: "Formatted at 2026-09-25 04:44:42 on dist-test-slave-kghz"
server_key: "33630ecea24d8515f015f5e7e75a4179"
server_key_iv: "c11bf6845888a623dc8db0a6431aaea6"
server_key_version: "encryptionkey@0"
I20260925 04:44:42.693956 6578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-0/wal/instance:
uuid: "0bbea834c4b244998ef191efd5bb4173"
format_stamp: "Formatted at 2026-09-25 04:44:42 on dist-test-slave-kghz"
server_key: "33630ecea24d8515f015f5e7e75a4179"
server_key_iv: "c11bf6845888a623dc8db0a6431aaea6"
server_key_version: "encryptionkey@0"
I20260925 04:44:42.697851 6578 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260925 04:44:42.700426 6593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:42.701643 6578 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 04:44:42.701781 6578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-0/wal
uuid: "0bbea834c4b244998ef191efd5bb4173"
format_stamp: "Formatted at 2026-09-25 04:44:42 on dist-test-slave-kghz"
server_key: "33630ecea24d8515f015f5e7e75a4179"
server_key_iv: "c11bf6845888a623dc8db0a6431aaea6"
server_key_version: "encryptionkey@0"
I20260925 04:44:42.701898 6578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:44:42.714186 6578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:44:42.717413 6578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:44:42.717635 6578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:44:42.718326 6578 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:44:42.719169 6578 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:44:42.719241 6578 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:42.719314 6578 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:44:42.719357 6578 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:42.728411 6578 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:46321
I20260925 04:44:42.728461 6706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:46321 every 8 connection(s)
I20260925 04:44:42.729391 6578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
I20260925 04:44:42.738073 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 6578
I20260925 04:44:42.738224 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-0/wal/instance
I20260925 04:44:42.738574 393 external_mini_cluster.cc:1428] Setting key 194924e48867af3fda3fdfcdcd706b53
Sep 25 04:44:42 dist-test-slave-kghz krb5kdc[6471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311482, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM
I20260925 04:44:42.744123 6555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:42.731348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:59311 (local address 127.0.98.126:46181)
0925 04:44:42.731503 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:42.731507 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:42.732363 (+ 856us) server_negotiation.cc:407] Connection header received
0925 04:44:42.733755 (+ 1392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:42.733758 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:42.733816 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:42.733925 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:42.735207 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:42.735764 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:42.736930 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:42.737171 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:42.737289 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:42.740305 (+ 3016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:42.740310 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:42.740317 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:42.740340 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:42.741906 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:42.742559 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:42.742564 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:42.742566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:42.742649 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:42.743010 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:42.743019 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:42.743020 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:42.743195 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:42.743298 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:42.743817 (+ 519us) server_negotiation.cc:299] Negotiation successful
0925 04:44:42.743945 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 04:44:42.744961 6709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:42.731730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:46181 (local address 127.0.98.65:59311)
0925 04:44:42.732202 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:42.732248 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:42.733523 (+ 1275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:42.734117 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:42.734125 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:42.734574 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:42.735056 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:42.735073 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:42.735894 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:42.735900 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:42.736793 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:42.736801 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:42.737330 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:42.737334 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:42.737506 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:42.738337 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:42.738360 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:42.740140 (+ 1780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:44:42.742073 (+ 1933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:42.742079 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:42.742095 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:42.742449 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:42.742746 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:42.742749 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:42.742751 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:42.742898 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:42.743307 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:44:42.743316 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:44:42.743577 (+ 261us) client_negotiation.cc:769] Sending connection context
0925 04:44:42.743767 (+ 190us) client_negotiation.cc:240] Negotiation successful
0925 04:44:42.744104 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":120,"threads_started":1}
I20260925 04:44:42.746291 6707 heartbeater.cc:347] Connected to a master server at 127.0.98.126:46181
I20260925 04:44:42.746555 6707 heartbeater.cc:464] Registering TS with master...
I20260925 04:44:42.747071 6707 heartbeater.cc:511] Master 127.0.98.126:46181 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.0.98.66@KRBTEST.COM; defaulting to no policy
I20260925 04:44:42.748696 6516 ts_manager.cc:194] Registered new tserver with Master: 0bbea834c4b244998ef191efd5bb4173 (127.0.98.65:46321)
I20260925 04:44:42.749925 6516 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.65@KRBTEST.COM'} at 127.0.98.65:59311
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/krb5kdc/oryx_127.0.98.66.keytab.
Entry for principal oryx/127.0.98.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/krb5kdc/oryx_127.0.98.66.keytab.
I20260925 04:44:42.766824 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:0
--local_ip_for_outbound_sockets=127.0.98.66
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:46181
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/krb5kdc/oryx_127.0.98.66.keytab
--principal=oryx/127.0.98.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:33935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:44:42.879566 6712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:44:42.879822 6712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:44:42.879868 6712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:44:42.884102 6712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:44:42.884166 6712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:44:42.884227 6712 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260925 04:44:42.889971 6712 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:33935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/krb5kdc/oryx_127.0.98.66.keytab
--principal=oryx/127.0.98.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:46181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6712
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:44:42.891116 6712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:44:42.892161 6712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:44:42.902588 6720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:42.902693 6717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:42.902675 6712 server_base.cc:1034] running on GCE node
W20260925 04:44:42.902758 6718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:42.903327 6712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:44:42.904039 6712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:44:42.905210 6712 hybrid_clock.cc:648] HybridClock initialized: now 1790311482905194 us; error 34 us; skew 500 ppm
Sep 25 04:44:42 dist-test-slave-kghz krb5kdc[6471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311482, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:44:42.908293 6712 init.cc:377] Logged in from keytab as oryx/127.0.98.66@KRBTEST.COM (short username oryx)
I20260925 04:44:42.909260 6712 webserver.cc:460] Webserver started at http://127.0.98.66:41511/ using document root <none> and password file <none>
I20260925 04:44:42.909842 6712 fs_manager.cc:359] Metadata directory not provided
I20260925 04:44:42.909943 6712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:44:42.910188 6712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:44:42.912372 6712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-1/data/instance:
uuid: "49d951568c0a4ae2a35eeedd7daf1646"
format_stamp: "Formatted at 2026-09-25 04:44:42 on dist-test-slave-kghz"
server_key: "53880ea9d4b1d0d760d09a53b696d2ad"
server_key_iv: "142fb452b461d591bf7355e9b4bf7354"
server_key_version: "encryptionkey@0"
I20260925 04:44:42.912979 6712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-1/wal/instance:
uuid: "49d951568c0a4ae2a35eeedd7daf1646"
format_stamp: "Formatted at 2026-09-25 04:44:42 on dist-test-slave-kghz"
server_key: "53880ea9d4b1d0d760d09a53b696d2ad"
server_key_iv: "142fb452b461d591bf7355e9b4bf7354"
server_key_version: "encryptionkey@0"
I20260925 04:44:42.916968 6712 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 04:44:42.919579 6727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:42.920915 6712 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 04:44:42.921108 6712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-1/wal
uuid: "49d951568c0a4ae2a35eeedd7daf1646"
format_stamp: "Formatted at 2026-09-25 04:44:42 on dist-test-slave-kghz"
server_key: "53880ea9d4b1d0d760d09a53b696d2ad"
server_key_iv: "142fb452b461d591bf7355e9b4bf7354"
server_key_version: "encryptionkey@0"
I20260925 04:44:42.921245 6712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:44:42.934818 6712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:44:42.938313 6712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:44:42.938522 6712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:44:42.939114 6712 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:44:42.939962 6712 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:44:42.940032 6712 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:42.940099 6712 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:44:42.940147 6712 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:42.949515 6712 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:36667
I20260925 04:44:42.949570 6840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:36667 every 8 connection(s)
I20260925 04:44:42.950575 6712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
I20260925 04:44:42.953677 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 6712
I20260925 04:44:42.953831 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-1/wal/instance
I20260925 04:44:42.954174 393 external_mini_cluster.cc:1428] Setting key 79a22483fe9bfafd4afab0799cbcf887
Sep 25 04:44:42 dist-test-slave-kghz krb5kdc[6471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311482, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.98.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.67@KRBTEST.COM" created.
I20260925 04:44:42.967139 6555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:42.952651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:43211 (local address 127.0.98.126:46181)
0925 04:44:42.952819 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:42.952824 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:42.953844 (+ 1020us) server_negotiation.cc:407] Connection header received
0925 04:44:42.955709 (+ 1865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:42.955713 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:42.955761 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:42.955840 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:42.957063 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:42.957939 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:42.959312 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:42.959733 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:42.959864 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:42.962827 (+ 2963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:42.962830 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:42.962844 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:42.962870 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:42.964804 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:42.965453 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:42.965461 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:42.965463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:42.965547 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:42.965929 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:42.965932 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:42.965934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:42.966127 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:42.966236 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:42.966847 (+ 611us) server_negotiation.cc:299] Negotiation successful
0925 04:44:42.966973 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 04:44:42.968001 6843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:42.952991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:46181 (local address 127.0.98.66:43211)
0925 04:44:42.953650 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:42.953715 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:42.955458 (+ 1743us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:42.956022 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:42.956030 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:42.956395 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:42.956886 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:42.956919 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:42.958111 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:42.958115 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:42.959174 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:42.959185 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:42.959919 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:42.959923 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:42.960086 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:42.960797 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:42.960815 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:42.962574 (+ 1759us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:44:42.964949 (+ 2375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:42.964955 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:42.964973 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:42.965330 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:42.965647 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:42.965650 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:42.965653 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:42.965835 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:42.966264 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:44:42.966272 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:44:42.966582 (+ 310us) client_negotiation.cc:769] Sending connection context
0925 04:44:42.966790 (+ 208us) client_negotiation.cc:240] Negotiation successful
0925 04:44:42.967151 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":206,"threads_started":1}
I20260925 04:44:42.969321 6841 heartbeater.cc:347] Connected to a master server at 127.0.98.126:46181
I20260925 04:44:42.969585 6841 heartbeater.cc:464] Registering TS with master...
I20260925 04:44:42.970110 6841 heartbeater.cc:511] Master 127.0.98.126:46181 requested a full tablet report, sending...
I20260925 04:44:42.971225 6516 ts_manager.cc:194] Registered new tserver with Master: 49d951568c0a4ae2a35eeedd7daf1646 (127.0.98.66:36667)
I20260925 04:44:42.971800 6516 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.66@KRBTEST.COM'} at 127.0.98.66:43211
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/krb5kdc/oryx_127.0.98.67.keytab.
Entry for principal oryx/127.0.98.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/krb5kdc/oryx_127.0.98.67.keytab.
I20260925 04:44:42.985342 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:0
--local_ip_for_outbound_sockets=127.0.98.67
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:46181
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/krb5kdc/oryx_127.0.98.67.keytab
--principal=oryx/127.0.98.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:33935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:44:43.099210 6846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:44:43.099503 6846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:44:43.099550 6846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:44:43.103715 6846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:44:43.103776 6846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:44:43.103834 6846 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260925 04:44:43.109205 6846 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:33935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/krb5kdc/oryx_127.0.98.67.keytab
--principal=oryx/127.0.98.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:46181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6846
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:44:43.110363 6846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:44:43.111421 6846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:44:43.121384 6851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:43.121493 6852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:43.121459 6846 server_base.cc:1034] running on GCE node
W20260925 04:44:43.121415 6854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:43.122035 6846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:44:43.122766 6846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:44:43.123974 6846 hybrid_clock.cc:648] HybridClock initialized: now 1790311483123946 us; error 46 us; skew 500 ppm
Sep 25 04:44:43 dist-test-slave-kghz krb5kdc[6471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311483, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:44:43.127115 6846 init.cc:377] Logged in from keytab as oryx/127.0.98.67@KRBTEST.COM (short username oryx)
I20260925 04:44:43.128141 6846 webserver.cc:460] Webserver started at http://127.0.98.67:42029/ using document root <none> and password file <none>
I20260925 04:44:43.128733 6846 fs_manager.cc:359] Metadata directory not provided
I20260925 04:44:43.128826 6846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:44:43.129074 6846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:44:43.131325 6846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-2/data/instance:
uuid: "1870b79591534da9a311af3678fc937e"
format_stamp: "Formatted at 2026-09-25 04:44:43 on dist-test-slave-kghz"
server_key: "72dab39a63b15b8f02ecc71a8e7025c9"
server_key_iv: "e85a9c7847903b53e8bbc6875921bd1e"
server_key_version: "encryptionkey@0"
I20260925 04:44:43.131968 6846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-2/wal/instance:
uuid: "1870b79591534da9a311af3678fc937e"
format_stamp: "Formatted at 2026-09-25 04:44:43 on dist-test-slave-kghz"
server_key: "72dab39a63b15b8f02ecc71a8e7025c9"
server_key_iv: "e85a9c7847903b53e8bbc6875921bd1e"
server_key_version: "encryptionkey@0"
I20260925 04:44:43.136040 6846 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 04:44:43.138777 6861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:43.140115 6846 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:44:43.140275 6846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-2/wal
uuid: "1870b79591534da9a311af3678fc937e"
format_stamp: "Formatted at 2026-09-25 04:44:43 on dist-test-slave-kghz"
server_key: "72dab39a63b15b8f02ecc71a8e7025c9"
server_key_iv: "e85a9c7847903b53e8bbc6875921bd1e"
server_key_version: "encryptionkey@0"
I20260925 04:44:43.140405 6846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:44:43.154793 6846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:44:43.158291 6846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:44:43.158511 6846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:44:43.159109 6846 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:44:43.159921 6846 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:44:43.159992 6846 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:43.160064 6846 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:44:43.160108 6846 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:43.169636 6846 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:38689
I20260925 04:44:43.169667 6974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:38689 every 8 connection(s)
I20260925 04:44:43.170590 6846 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
I20260925 04:44:43.172459 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 6846
I20260925 04:44:43.172595 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-2/wal/instance
I20260925 04:44:43.172887 393 external_mini_cluster.cc:1428] Setting key 58f099b0499b71a528c6ed30a45a0fe3
Sep 25 04:44:43 dist-test-slave-kghz krb5kdc[6471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311483, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM
I20260925 04:44:43.185074 6555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:43.172401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:52487 (local address 127.0.98.126:46181)
0925 04:44:43.172576 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:43.172582 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:43.173708 (+ 1126us) server_negotiation.cc:407] Connection header received
0925 04:44:43.175245 (+ 1537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:43.175249 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:43.175302 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:43.175380 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:43.176538 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:43.177087 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:43.178277 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:43.178581 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:43.178724 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:43.181264 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:43.181267 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:43.181270 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:43.181295 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:43.182886 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:43.183351 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:43.183354 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:43.183355 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:43.183429 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:43.183925 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:43.183928 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:43.183929 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:43.184128 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:43.184218 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:43.184798 (+ 580us) server_negotiation.cc:299] Negotiation successful
0925 04:44:43.184929 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 04:44:43.185843 6977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:43.172853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:46181 (local address 127.0.98.67:52487)
0925 04:44:43.173561 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:43.173616 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:43.175034 (+ 1418us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:43.175550 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:43.175557 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:43.175909 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:43.176393 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:43.176406 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:43.177228 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:43.177231 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:43.178128 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:43.178136 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:43.178760 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:43.178764 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:43.178947 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:43.179557 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:43.179574 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:43.181104 (+ 1530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:44:43.182988 (+ 1884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:43.182995 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:43.183009 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:43.183261 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:43.183521 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:43.183524 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:43.183525 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:43.183648 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:43.184257 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:44:43.184263 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:44:43.184525 (+ 262us) client_negotiation.cc:769] Sending connection context
0925 04:44:43.184717 (+ 192us) client_negotiation.cc:240] Negotiation successful
0925 04:44:43.184979 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":145,"threads_started":1}
I20260925 04:44:43.187140 6975 heartbeater.cc:347] Connected to a master server at 127.0.98.126:46181
I20260925 04:44:43.187407 6975 heartbeater.cc:464] Registering TS with master...
I20260925 04:44:43.187981 6975 heartbeater.cc:511] Master 127.0.98.126:46181 requested a full tablet report, sending...
I20260925 04:44:43.189007 6516 ts_manager.cc:194] Registered new tserver with Master: 1870b79591534da9a311af3678fc937e (127.0.98.67:38689)
I20260925 04:44:43.189576 6516 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.67@KRBTEST.COM'} at 127.0.98.67:52487
I20260925 04:44:43.199251 393 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/krb5kdc/oryx_127.0.98.125.keytab.
Entry for principal oryx/127.0.98.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/krb5kdc/oryx_127.0.98.125.keytab.
I20260925 04:44:43.229491 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.125:45861
--webserver_interface=127.0.98.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/krb5kdc/oryx_127.0.98.125.keytab
--principal=oryx/127.0.98.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:33935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:46181,127.0.98.125:45861
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:44:43.342394 6980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:44:43.342691 6980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:44:43.342736 6980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:44:43.346993 6980 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:44:43.347060 6980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:44:43.347080 6980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:44:43.347097 6980 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:44:43.347113 6980 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:44:43.352502 6980 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:33935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:46181,127.0.98.125:45861
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/krb5kdc/oryx_127.0.98.125.keytab
--principal=oryx/127.0.98.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.125:45861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6980
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
W20260925 04:44:43.352737 6980 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.126:46181,127.0.98.125:45861'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260925 04:44:43.353652 6980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:44:43.354701 6980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:44:43.363543 6985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:43.363530 6986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:43.363530 6988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:43.363581 6980 server_base.cc:1034] running on GCE node
I20260925 04:44:43.364529 6980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:44:43.365746 6980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:44:43.366907 6980 hybrid_clock.cc:648] HybridClock initialized: now 1790311483366857 us; error 71 us; skew 500 ppm
Sep 25 04:44:43 dist-test-slave-kghz krb5kdc[6471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311483, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:44:43.369881 6980 init.cc:377] Logged in from keytab as oryx/127.0.98.125@KRBTEST.COM (short username oryx)
I20260925 04:44:43.370911 6980 webserver.cc:460] Webserver started at http://127.0.98.125:43565/ using document root <none> and password file <none>
I20260925 04:44:43.371500 6980 fs_manager.cc:359] Metadata directory not provided
I20260925 04:44:43.371564 6980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:44:43.371811 6980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:44:43.374151 6980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/master-1/data/instance:
uuid: "4f99000937494cc99e7b9bd62901f3f0"
format_stamp: "Formatted at 2026-09-25 04:44:43 on dist-test-slave-kghz"
server_key: "d129d18a1a8b3b22d102eeb9315cc812"
server_key_iv: "7a8d1b6c2dae08a6d2445a3e8921b6c9"
server_key_version: "encryptionkey@0"
I20260925 04:44:43.374823 6980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/master-1/wal/instance:
uuid: "4f99000937494cc99e7b9bd62901f3f0"
format_stamp: "Formatted at 2026-09-25 04:44:43 on dist-test-slave-kghz"
server_key: "d129d18a1a8b3b22d102eeb9315cc812"
server_key_iv: "7a8d1b6c2dae08a6d2445a3e8921b6c9"
server_key_version: "encryptionkey@0"
I20260925 04:44:43.378662 6980 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 04:44:43.381153 6995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:43.382478 6980 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 04:44:43.382606 6980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/master-1/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/master-1/wal
uuid: "4f99000937494cc99e7b9bd62901f3f0"
format_stamp: "Formatted at 2026-09-25 04:44:43 on dist-test-slave-kghz"
server_key: "d129d18a1a8b3b22d102eeb9315cc812"
server_key_iv: "7a8d1b6c2dae08a6d2445a3e8921b6c9"
server_key_version: "encryptionkey@0"
I20260925 04:44:43.382712 6980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:44:43.399505 6980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:44:43.402894 6980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:44:43.403100 6980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:44:43.410235 6980 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.125:45861
I20260925 04:44:43.410543 7047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.125:45861 every 8 connection(s)
I20260925 04:44:43.411350 6980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/master-1/data/info.pb
I20260925 04:44:43.414193 7048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:44:43.416079 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 6980
I20260925 04:44:43.419379 7048 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46181 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 04:44:43 dist-test-slave-kghz krb5kdc[6471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.125@KRBTEST.COM
Sep 25 04:44:43 dist-test-slave-kghz krb5kdc[6471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311483, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.125@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM
I20260925 04:44:43.431028 6555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:43.421274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39346 (local address 127.0.98.126:46181)
0925 04:44:43.421413 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:43.421417 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:43.421764 (+ 347us) server_negotiation.cc:407] Connection header received
0925 04:44:43.421886 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:43.421890 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:43.421947 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:43.422038 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:43.422832 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:43.423540 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:43.424675 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:43.424882 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:43.424989 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:43.427572 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:43.427578 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:43.427582 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:43.427617 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:43.429138 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:43.429593 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:43.429597 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:43.429599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:43.429687 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:43.429998 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:43.430002 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:43.430003 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:43.430231 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:43.430373 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:43.430726 (+ 353us) server_negotiation.cc:299] Negotiation successful
0925 04:44:43.430855 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 04:44:43.431845 7051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:43.417741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49406 (local address 127.0.98.125:45861)
0925 04:44:43.418197 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:43.418218 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:43.418291 (+ 73us) server_negotiation.cc:407] Connection header received
0925 04:44:43.419148 (+ 857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:43.419169 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:43.419579 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:43.419933 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:43.420643 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:43.421893 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:43.423107 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:43.423785 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:43.423894 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:43.426742 (+ 2848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:43.426747 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:43.426757 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:43.426778 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:43.428821 (+ 2043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:43.429310 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:43.429321 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:43.429322 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:43.429431 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:43.429708 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:43.429713 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:43.429715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:43.430101 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:43.430344 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:43.430605 (+ 261us) server_negotiation.cc:299] Negotiation successful
0925 04:44:43.430971 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":261,"thread_start_us":107,"threads_started":1}
I20260925 04:44:43.431843 7052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:43.421223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:46181 (local address 127.0.0.1:39346)
0925 04:44:43.421628 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:43.421676 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:43.421792 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:43.422089 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:43.422095 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:43.422562 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:43.422710 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:43.422716 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:43.423667 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:43.423670 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:43.424552 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:43.424564 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:43.425054 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:43.425082 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:43.425229 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:43.425850 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:43.425867 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:43.427417 (+ 1550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:44:43.429245 (+ 1828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:43.429255 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:43.429265 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:43.429485 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:43.429781 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:43.429786 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:43.429787 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:43.429882 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:43.430371 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:44:43.430378 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:44:43.430516 (+ 138us) client_negotiation.cc:769] Sending connection context
0925 04:44:43.430660 (+ 144us) client_negotiation.cc:240] Negotiation successful
0925 04:44:43.431049 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":139,"thread_start_us":160,"threads_started":1}
I20260925 04:44:43.433318 7048 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.125" port: 45861 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 04:44:43 dist-test-slave-kghz krb5kdc[6471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311483, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.125@KRBTEST.COM for oryx/127.0.98.125@KRBTEST.COM
I20260925 04:44:43.443342 7052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:43.433928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:45861 (local address 127.0.0.1:49410)
0925 04:44:43.434211 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:43.434224 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:43.434372 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:43.434595 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:43.434597 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:43.434739 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:43.434834 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:43.434846 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:43.435557 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:43.435569 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:43.436165 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:43.436173 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:43.436647 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:43.436650 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:43.436735 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:43.437441 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:43.437456 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:43.439724 (+ 2268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:44:43.441770 (+ 2046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:43.441773 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:43.441775 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:43.441985 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:43.442343 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:43.442351 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:43.442353 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:43.442435 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:43.442892 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:44:43.442895 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:44:43.443012 (+ 117us) client_negotiation.cc:769] Sending connection context
0925 04:44:43.443098 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 04:44:43.443210 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221}
I20260925 04:44:43.443483 7051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:43.433958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49410 (local address 127.0.98.125:45861)
0925 04:44:43.434059 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:43.434063 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:43.434345 (+ 282us) server_negotiation.cc:407] Connection header received
0925 04:44:43.434450 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:43.434453 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:43.434494 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:43.434566 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:43.434932 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:43.435451 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:43.436295 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:43.436521 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:43.436624 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:43.439876 (+ 3252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:43.439880 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:43.439882 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:43.439911 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:43.441602 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:43.442091 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:43.442095 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:43.442097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:43.442184 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:43.442537 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:43.442540 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:43.442541 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:43.442768 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:43.442859 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:43.443183 (+ 324us) server_negotiation.cc:299] Negotiation successful
0925 04:44:43.443296 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 04:44:43.446532 7048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f99000937494cc99e7b9bd62901f3f0: Bootstrap starting.
I20260925 04:44:43.448776 7048 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f99000937494cc99e7b9bd62901f3f0: Neither blocks nor log segments found. Creating new log.
I20260925 04:44:43.449524 7048 log.cc:824] T 00000000000000000000000000000000 P 4f99000937494cc99e7b9bd62901f3f0: Log is configured to *not* fsync() on all Append() calls
I20260925 04:44:43.451370 7048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f99000937494cc99e7b9bd62901f3f0: No bootstrap required, opened a new log
I20260925 04:44:43.454358 7048 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f99000937494cc99e7b9bd62901f3f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58edb7db5329430685f0d9e7a5098394" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46181 } } peers { permanent_uuid: "4f99000937494cc99e7b9bd62901f3f0" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 45861 } }
I20260925 04:44:43.454564 7048 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4f99000937494cc99e7b9bd62901f3f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:44:43.454638 7048 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f99000937494cc99e7b9bd62901f3f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f99000937494cc99e7b9bd62901f3f0, State: Initialized, Role: FOLLOWER
I20260925 04:44:43.455098 7048 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f99000937494cc99e7b9bd62901f3f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58edb7db5329430685f0d9e7a5098394" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46181 } } peers { permanent_uuid: "4f99000937494cc99e7b9bd62901f3f0" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 45861 } }
I20260925 04:44:43.455574 7054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f99000937494cc99e7b9bd62901f3f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58edb7db5329430685f0d9e7a5098394" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46181 } } peers { permanent_uuid: "4f99000937494cc99e7b9bd62901f3f0" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 45861 } } }
I20260925 04:44:43.455682 7054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f99000937494cc99e7b9bd62901f3f0 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:44:43.456274 7048 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f99000937494cc99e7b9bd62901f3f0 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 04:44:43.460973 7065 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4f99000937494cc99e7b9bd62901f3f0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 04:44:43.461055 7065 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 04:44:43.463564 6980 master_runner.cc:428] Detected that this master 4f99000937494cc99e7b9bd62901f3f0 is joining an existing cluster
I20260925 04:44:43.463630 6980 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.125:45861
I20260925 04:44:43.475602 7072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:43.466110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:45861 (local address 127.0.0.1:49426)
0925 04:44:43.466558 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:43.466573 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:43.466748 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:43.467254 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:43.467258 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:43.467501 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:43.467632 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:43.467641 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:43.468577 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:43.468580 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:43.469762 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:43.469772 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:43.470467 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:43.470471 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:43.470585 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:43.471158 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:43.471169 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:43.471617 (+ 448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:44:43.473931 (+ 2314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:43.473935 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:43.473938 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:43.474184 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:43.474491 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:43.474495 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:43.474497 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:43.474570 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:43.475017 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:44:43.475020 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:44:43.475112 (+ 92us) client_negotiation.cc:769] Sending connection context
0925 04:44:43.475244 (+ 132us) client_negotiation.cc:240] Negotiation successful
0925 04:44:43.475397 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"mutex_wait_us":156,"thread_start_us":78,"threads_started":1}
I20260925 04:44:43.475715 6555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:43.465936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39358 (local address 127.0.98.126:46181)
0925 04:44:43.466063 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:43.466067 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:43.466459 (+ 392us) server_negotiation.cc:407] Connection header received
0925 04:44:43.466668 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:43.466672 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:43.466730 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:43.466786 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:43.467550 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:43.468354 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:43.470338 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:43.470644 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:43.470769 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:43.472335 (+ 1566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:43.472343 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:43.472345 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:43.472368 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:43.473973 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:43.474452 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:43.474458 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:43.474460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:43.474552 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:43.474859 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:43.474863 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:43.474865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:43.475091 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:43.475190 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:43.475456 (+ 266us) server_negotiation.cc:299] Negotiation successful
0925 04:44:43.475572 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 04:44:43.475628 7051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:43.465925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49426 (local address 127.0.98.125:45861)
0925 04:44:43.466037 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:43.466042 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:43.466667 (+ 625us) server_negotiation.cc:407] Connection header received
0925 04:44:43.466878 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:43.466881 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:43.466959 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:43.467037 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:43.467755 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:43.468467 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:43.469890 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:43.470329 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:43.470461 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:43.471783 (+ 1322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:43.471788 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:43.471790 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:43.471813 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:43.473815 (+ 2002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:43.474288 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:43.474294 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:43.474298 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:43.474378 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:43.474674 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:43.474677 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:43.474679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:43.474916 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:43.475014 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:43.475291 (+ 277us) server_negotiation.cc:299] Negotiation successful
0925 04:44:43.475469 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":41}
I20260925 04:44:43.476336 7071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:43.465863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:46181 (local address 127.0.0.1:39358)
0925 04:44:43.466367 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:43.466387 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:43.466537 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:43.466960 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:43.466964 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:43.467273 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:43.467428 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:43.467434 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:43.468475 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:43.468478 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:43.469762 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:43.469772 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:43.471118 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:43.471126 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:43.471217 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:43.471716 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:43.471732 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:43.472195 (+ 463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:44:43.474087 (+ 1892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:43.474091 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:43.474093 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:43.474352 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:43.474650 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:43.474654 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:43.474656 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:43.474759 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:43.475190 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:44:43.475193 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:44:43.475283 (+ 90us) client_negotiation.cc:769] Sending connection context
0925 04:44:43.475894 (+ 611us) client_negotiation.cc:240] Negotiation successful
0925 04:44:43.476055 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"mutex_wait_us":160,"thread_start_us":127,"threads_started":1}
Sep 25 04:44:43 dist-test-slave-kghz krb5kdc[6471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311482, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.126@KRBTEST.COM for oryx/127.0.98.125@KRBTEST.COM
I20260925 04:44:43.491909 7073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:43.480539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:45861 (local address 127.0.0.1:49428)
0925 04:44:43.482543 (+ 2004us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:43.482605 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:43.482745 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:43.483093 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:43.483099 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:43.483547 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:43.483676 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:43.483683 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:43.484509 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:43.484512 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:43.485386 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:43.485396 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:43.486123 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:43.486130 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:43.486304 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:43.486938 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:43.486960 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:43.488560 (+ 1600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:44:43.490233 (+ 1673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:43.490242 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:43.490270 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:43.490518 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:43.490806 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:43.490810 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:43.490812 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:43.490901 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:43.491284 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:44:43.491292 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:44:43.491429 (+ 137us) client_negotiation.cc:769] Sending connection context
0925 04:44:43.491568 (+ 139us) client_negotiation.cc:240] Negotiation successful
0925 04:44:43.491756 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1819,"thread_start_us":79,"threads_started":1}
I20260925 04:44:43.491904 7051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:43.480617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49428 (local address 127.0.98.125:45861)
0925 04:44:43.480725 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:43.480729 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:43.482707 (+ 1978us) server_negotiation.cc:407] Connection header received
0925 04:44:43.482873 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:43.482875 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:43.482956 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:43.483030 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:43.483792 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:43.484386 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:43.485567 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:43.485905 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:43.486085 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:43.488710 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:43.488713 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:43.488716 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:43.488739 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:43.490101 (+ 1362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:43.490622 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:43.490627 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:43.490629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:43.490711 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:43.491004 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:43.491007 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:43.491008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:43.491180 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:43.491284 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:43.491633 (+ 349us) server_negotiation.cc:299] Negotiation successful
0925 04:44:43.491746 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 04:44:43.493736 6516 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.125:45861: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "4f99000937494cc99e7b9bd62901f3f0"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.125"
port: 45861
}
attrs {
promote: true
}
}
dest_uuid: "58edb7db5329430685f0d9e7a5098394"
cas_config_opid_index: -1
I20260925 04:44:43.494441 6516 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58edb7db5329430685f0d9e7a5098394 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "58edb7db5329430685f0d9e7a5098394" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46181 } } peers { permanent_uuid: "4f99000937494cc99e7b9bd62901f3f0" member_type: NON_VOTER last_known_addr { host: "127.0.98.125" port: 45861 } attrs { promote: true } }
WARNING: no policy specified for oryx/127.0.98.124@KRBTEST.COM; defaulting to no policy
I20260925 04:44:43.496042 7076 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 58edb7db5329430685f0d9e7a5098394 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 4f99000937494cc99e7b9bd62901f3f0 (127.0.98.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "58edb7db5329430685f0d9e7a5098394" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46181 } } peers { permanent_uuid: "4f99000937494cc99e7b9bd62901f3f0" member_type: NON_VOTER last_known_addr { host: "127.0.98.125" port: 45861 } attrs { promote: true } } }
I20260925 04:44:43.496779 7022 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f99000937494cc99e7b9bd62901f3f0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:44:43.497509 7076 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.125:45861
I20260925 04:44:43.497741 7075 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58edb7db5329430685f0d9e7a5098394 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "58edb7db5329430685f0d9e7a5098394" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "58edb7db5329430685f0d9e7a5098394" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46181 } } peers { permanent_uuid: "4f99000937494cc99e7b9bd62901f3f0" member_type: NON_VOTER last_known_addr { host: "127.0.98.125" port: 45861 } attrs { promote: true } } }
I20260925 04:44:43.497871 7075 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58edb7db5329430685f0d9e7a5098394 [sys.catalog]: This master's current role is: LEADER
I20260925 04:44:43.497956 6980 master.cc:537] Master@127.0.98.125:45861 shutting down...
I20260925 04:44:43.498030 7022 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4f99000937494cc99e7b9bd62901f3f0 [term 1 FOLLOWER]: Refusing update from remote peer 58edb7db5329430685f0d9e7a5098394: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260925 04:44:43.498780 7075 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 58edb7db5329430685f0d9e7a5098394 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f99000937494cc99e7b9bd62901f3f0" member_type: NON_VOTER last_known_addr { host: "127.0.98.125" port: 45861 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 04:44:43.500837 6501 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260925 04:44:43.501799 6501 consensus_peers.cc:487] T 00000000000000000000000000000000 P 58edb7db5329430685f0d9e7a5098394 -> Peer 4f99000937494cc99e7b9bd62901f3f0 (127.0.98.125:45861): Couldn't send request to peer 4f99000937494cc99e7b9bd62901f3f0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 04:44:43.502743 6980 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 4f99000937494cc99e7b9bd62901f3f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:44:43.503027 6980 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 4f99000937494cc99e7b9bd62901f3f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:44:43.503131 6980 tablet_replica.cc:330] T 00000000000000000000000000000000 P 4f99000937494cc99e7b9bd62901f3f0: stopping tablet replica
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.124@KRBTEST.COM" created.
I20260925 04:44:43.520813 6980 master.cc:552] Master@127.0.98.125:45861 shutdown complete.
I20260925 04:44:43.520874 6980 master_runner.cc:305] Clearing existing system tablet
I20260925 04:44:43.521222 6980 fs_manager.cc:359] Metadata directory not provided
I20260925 04:44:43.521281 6980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:44:43.523160 6980 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 04:44:43.524040 7081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:43.524259 6980 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:44:43.524394 6980 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 04:44:43.524497 6980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/master-1/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/master-1/wal
uuid: "4f99000937494cc99e7b9bd62901f3f0"
format_stamp: "Formatted at 2026-09-25 04:44:43 on dist-test-slave-kghz"
server_key: "d129d18a1a8b3b22d102eeb9315cc812"
server_key_iv: "7a8d1b6c2dae08a6d2445a3e8921b6c9"
server_key_version: "encryptionkey@0"
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/krb5kdc/oryx_127.0.98.124.keytab.
Entry for principal oryx/127.0.98.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/krb5kdc/oryx_127.0.98.124.keytab.
I20260925 04:44:43.525574 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.124:41887
--webserver_interface=127.0.98.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/krb5kdc/oryx_127.0.98.124.keytab
--principal=oryx/127.0.98.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:33935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:46181,127.0.98.125:45861,127.0.98.124:41887
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 04:44:43.526759 6980 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 4f99000937494cc99e7b9bd62901f3f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:44:43.529110 6980 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 4f99000937494cc99e7b9bd62901f3f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 04:44:43.529204 6980 log.cc:1196] T 00000000000000000000000000000000 P 4f99000937494cc99e7b9bd62901f3f0: Deleting WAL directory at /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260925 04:44:43.529464 6980 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 4f99000937494cc99e7b9bd62901f3f0: Deleting consensus metadata
I20260925 04:44:43.529641 6980 master_runner.cc:315] Copying system tablet from 127.0.98.126:46181
I20260925 04:44:43.530843 6980 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 4f99000937494cc99e7b9bd62901f3f0: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.126:46181
I20260925 04:44:43.541126 6555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:43.531562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39360 (local address 127.0.98.126:46181)
0925 04:44:43.531683 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:43.531688 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:43.531988 (+ 300us) server_negotiation.cc:407] Connection header received
0925 04:44:43.532169 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:43.532173 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:43.532228 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:43.532311 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:43.533398 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:43.534228 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:43.535693 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:43.536032 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:43.536141 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:43.537555 (+ 1414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:43.537559 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:43.537562 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:43.537589 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:43.539405 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:43.539898 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:43.539901 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:43.539903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:43.539990 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:43.540267 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:43.540270 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:43.540272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:43.540489 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:43.540582 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:43.540853 (+ 271us) server_negotiation.cc:299] Negotiation successful
0925 04:44:43.540970 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 04:44:43.541131 7088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:43.531481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:46181 (local address 127.0.0.1:39360)
0925 04:44:43.531861 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:43.531880 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:43.532029 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:43.532358 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:43.532364 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:43.532639 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:43.533236 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:43.533251 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:43.534403 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:43.534410 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:43.535545 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:43.535560 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:43.536171 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:43.536175 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:43.536308 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:43.536949 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:43.536964 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:43.537400 (+ 436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:44:43.539533 (+ 2133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:43.539537 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:43.539539 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:43.539797 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:43.540084 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:43.540087 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:43.540089 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:43.540180 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:43.540582 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:44:43.540584 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:44:43.540684 (+ 100us) client_negotiation.cc:769] Sending connection context
0925 04:44:43.540786 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 04:44:43.540932 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":96,"threads_started":1}
I20260925 04:44:43.542109 6536 tablet_copy_service.cc:140] P 58edb7db5329430685f0d9e7a5098394: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 4f99000937494cc99e7b9bd62901f3f0 ({username='oryx', principal='oryx/127.0.98.125@KRBTEST.COM'} at 127.0.0.1:39360)
I20260925 04:44:43.542330 6536 tablet_copy_service.cc:161] P 58edb7db5329430685f0d9e7a5098394: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 4f99000937494cc99e7b9bd62901f3f0 at {username='oryx', principal='oryx/127.0.98.125@KRBTEST.COM'} at 127.0.0.1:39360: session id = 4f99000937494cc99e7b9bd62901f3f0-00000000000000000000000000000000
I20260925 04:44:43.544142 6536 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 58edb7db5329430685f0d9e7a5098394: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 04:44:43.545888 6980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 04:44:43.548300 6980 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 4f99000937494cc99e7b9bd62901f3f0: tablet copy: Starting download of 0 data blocks...
I20260925 04:44:43.548509 6980 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 4f99000937494cc99e7b9bd62901f3f0: tablet copy: Starting download of 1 WAL segments...
I20260925 04:44:43.550420 6980 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 4f99000937494cc99e7b9bd62901f3f0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 04:44:43.552477 6536 tablet_copy_service.cc:342] P 58edb7db5329430685f0d9e7a5098394: Request end of tablet copy session 4f99000937494cc99e7b9bd62901f3f0-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.125@KRBTEST.COM'} at 127.0.0.1:39360
I20260925 04:44:43.552618 6536 tablet_copy_service.cc:433] P 58edb7db5329430685f0d9e7a5098394: ending tablet copy session 4f99000937494cc99e7b9bd62901f3f0-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 4f99000937494cc99e7b9bd62901f3f0
I20260925 04:44:43.560443 6980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:44:43.562613 7090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:43.562662 6980 server_base.cc:1034] running on GCE node
W20260925 04:44:43.562721 7092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:43.563045 7089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:43.563362 6980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:44:43.563684 6980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:44:43.564842 6980 hybrid_clock.cc:648] HybridClock initialized: now 1790311483564831 us; error 29 us; skew 500 ppm
Sep 25 04:44:43 dist-test-slave-kghz krb5kdc[6471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311483, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:44:43.566643 6980 init.cc:377] Logged in from keytab as oryx/127.0.98.125@KRBTEST.COM (short username oryx)
I20260925 04:44:43.567133 6980 webserver.cc:460] Webserver started at http://127.0.98.125:40871/ using document root <none> and password file <none>
I20260925 04:44:43.567319 6980 fs_manager.cc:359] Metadata directory not provided
I20260925 04:44:43.567353 6980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:44:43.568758 6980 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 04:44:43.569813 7099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:43.570083 6980 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 04:44:43.570189 6980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/master-1/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/master-1/wal
uuid: "4f99000937494cc99e7b9bd62901f3f0"
format_stamp: "Formatted at 2026-09-25 04:44:43 on dist-test-slave-kghz"
server_key: "d129d18a1a8b3b22d102eeb9315cc812"
server_key_iv: "7a8d1b6c2dae08a6d2445a3e8921b6c9"
server_key_version: "encryptionkey@0"
I20260925 04:44:43.570384 6980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:44:43.601114 6980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:44:43.603616 6980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:44:43.609201 6980 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.125:45861
I20260925 04:44:43.609247 7151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.125:45861 every 8 connection(s)
I20260925 04:44:43.610097 6980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/master-1/data/info.pb
I20260925 04:44:43.612112 7152 sys_catalog.cc:263] Verifying existing consensus state
I20260925 04:44:43.613270 7152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f99000937494cc99e7b9bd62901f3f0: Bootstrap starting.
I20260925 04:44:43.625093 7152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f99000937494cc99e7b9bd62901f3f0: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 04:44:43.625475 7152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f99000937494cc99e7b9bd62901f3f0: Bootstrap complete.
I20260925 04:44:43.626159 7152 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f99000937494cc99e7b9bd62901f3f0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "58edb7db5329430685f0d9e7a5098394" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46181 } } peers { permanent_uuid: "4f99000937494cc99e7b9bd62901f3f0" member_type: NON_VOTER last_known_addr { host: "127.0.98.125" port: 45861 } attrs { promote: true } }
I20260925 04:44:43.626334 7152 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f99000937494cc99e7b9bd62901f3f0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4f99000937494cc99e7b9bd62901f3f0, State: Initialized, Role: LEARNER
I20260925 04:44:43.626474 7152 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f99000937494cc99e7b9bd62901f3f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "58edb7db5329430685f0d9e7a5098394" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46181 } } peers { permanent_uuid: "4f99000937494cc99e7b9bd62901f3f0" member_type: NON_VOTER last_known_addr { host: "127.0.98.125" port: 45861 } attrs { promote: true } }
I20260925 04:44:43.626894 7155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f99000937494cc99e7b9bd62901f3f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "58edb7db5329430685f0d9e7a5098394" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46181 } } peers { permanent_uuid: "4f99000937494cc99e7b9bd62901f3f0" member_type: NON_VOTER last_known_addr { host: "127.0.98.125" port: 45861 } attrs { promote: true } } }
I20260925 04:44:43.627095 7155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f99000937494cc99e7b9bd62901f3f0 [sys.catalog]: This master's current role is: LEARNER
I20260925 04:44:43.627092 7152 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f99000937494cc99e7b9bd62901f3f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:44:43.631170 7166 catalog_manager.cc:1259] Loaded cluster ID: 279c26e79db84d6982461caf8787f885
I20260925 04:44:43.631248 7166 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4f99000937494cc99e7b9bd62901f3f0: loading cluster ID for follower catalog manager: success
I20260925 04:44:43.633384 7166 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4f99000937494cc99e7b9bd62901f3f0: acquiring CA information for follower catalog manager: success
I20260925 04:44:43.634825 7166 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4f99000937494cc99e7b9bd62901f3f0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 04:44:43.660061 7082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:44:43.660362 7082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:44:43.660452 7082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:44:43.664762 7082 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:44:43.664844 7082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:44:43.664882 7082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:44:43.664938 7082 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:44:43.664978 7082 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:44:43.670414 7082 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:33935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:46181,127.0.98.125:45861,127.0.98.124:41887
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/krb5kdc/oryx_127.0.98.124.keytab
--principal=oryx/127.0.98.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.124:41887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7082
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:44:43.671733 7082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:44:43.672816 7082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:44:43.681963 7172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:43.681928 7171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:43.682083 7082 server_base.cc:1034] running on GCE node
W20260925 04:44:43.682286 7174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:43.683005 7082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:44:43.684638 7082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:44:43.685843 7082 hybrid_clock.cc:648] HybridClock initialized: now 1790311483685820 us; error 50 us; skew 500 ppm
Sep 25 04:44:43 dist-test-slave-kghz krb5kdc[6471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311483, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:44:43.688885 7082 init.cc:377] Logged in from keytab as oryx/127.0.98.124@KRBTEST.COM (short username oryx)
I20260925 04:44:43.689893 7082 webserver.cc:460] Webserver started at http://127.0.98.124:41087/ using document root <none> and password file <none>
I20260925 04:44:43.690486 7082 fs_manager.cc:359] Metadata directory not provided
I20260925 04:44:43.690555 7082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:44:43.690734 7082 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:44:43.692903 7082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/master-2/data/instance:
uuid: "b655e5719e73450cb120a45e38394900"
format_stamp: "Formatted at 2026-09-25 04:44:43 on dist-test-slave-kghz"
server_key: "d65be6603cd4045170caf0c5b676a0b2"
server_key_iv: "68735a17b89bc191cebba0866a05a7ec"
server_key_version: "encryptionkey@0"
I20260925 04:44:43.693435 7082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/master-2/wal/instance:
uuid: "b655e5719e73450cb120a45e38394900"
format_stamp: "Formatted at 2026-09-25 04:44:43 on dist-test-slave-kghz"
server_key: "d65be6603cd4045170caf0c5b676a0b2"
server_key_iv: "68735a17b89bc191cebba0866a05a7ec"
server_key_version: "encryptionkey@0"
I20260925 04:44:43.697309 7082 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 04:44:43.700038 7181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:43.701445 7082 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 04:44:43.701574 7082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/master-2/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/master-2/wal
uuid: "b655e5719e73450cb120a45e38394900"
format_stamp: "Formatted at 2026-09-25 04:44:43 on dist-test-slave-kghz"
server_key: "d65be6603cd4045170caf0c5b676a0b2"
server_key_iv: "68735a17b89bc191cebba0866a05a7ec"
server_key_version: "encryptionkey@0"
I20260925 04:44:43.701666 7082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:44:43.729511 7082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:44:43.732862 7082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:44:43.733085 7082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:44:43.740865 7082 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.124:41887
I20260925 04:44:43.741258 7233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.124:41887 every 8 connection(s)
I20260925 04:44:43.741919 7082 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/master-2/data/info.pb
I20260925 04:44:43.742221 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 7082
I20260925 04:44:43.745455 7234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:44:43.750032 7234 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46181 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 04:44:43 dist-test-slave-kghz krb5kdc[6471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.124@KRBTEST.COM
I20260925 04:44:43.752691 6707 heartbeater.cc:503] Master 127.0.98.126:46181 was elected leader, sending a full tablet report...
I20260925 04:44:43.758936 7236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:43.743911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45564 (local address 127.0.98.124:41887)
0925 04:44:43.744391 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:43.744411 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:43.744473 (+ 62us) server_negotiation.cc:407] Connection header received
0925 04:44:43.745190 (+ 717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:43.745208 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:43.745555 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:43.745894 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:43.746844 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:43.747848 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:43.749053 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:43.749596 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:43.749770 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:43.753200 (+ 3430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:43.753208 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:43.753226 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:43.753265 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:43.755558 (+ 2293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:43.756120 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:43.756126 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:43.756130 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:43.756247 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:43.756541 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:43.756544 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:43.756545 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:43.757179 (+ 634us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:43.757378 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:43.757732 (+ 354us) server_negotiation.cc:299] Negotiation successful
0925 04:44:43.758050 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":155,"threads_started":1}
Sep 25 04:44:43 dist-test-slave-kghz krb5kdc[6471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311483, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.124@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM
I20260925 04:44:43.763029 6555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:43.752220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39370 (local address 127.0.98.126:46181)
0925 04:44:43.752369 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:43.752374 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:43.753103 (+ 729us) server_negotiation.cc:407] Connection header received
0925 04:44:43.753316 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:43.753319 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:43.753372 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:43.753465 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:43.754433 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:43.755342 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:43.756635 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:43.756989 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:43.757110 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:43.759604 (+ 2494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:43.759608 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:43.759611 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:43.759640 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:43.761359 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:43.761792 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:43.761794 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:43.761795 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:43.761867 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:43.762119 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:43.762121 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:43.762122 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:43.762354 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:43.762432 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:43.762808 (+ 376us) server_negotiation.cc:299] Negotiation successful
0925 04:44:43.762904 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 04:44:43.763123 7237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:43.751993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:46181 (local address 127.0.0.1:39370)
0925 04:44:43.752455 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:43.752998 (+ 543us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:43.753201 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:43.753526 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:43.753534 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:43.754064 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:43.754298 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:43.754309 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:43.755487 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:43.755490 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:43.756515 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:43.756526 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:43.757130 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:43.757133 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:43.757228 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:43.757983 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:43.757995 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:43.759457 (+ 1462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:44:43.761455 (+ 1998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:43.761461 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:43.761465 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:43.761701 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:43.761947 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:43.761949 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:43.761951 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:43.762040 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:43.762445 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:44:43.762450 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:44:43.762567 (+ 117us) client_negotiation.cc:769] Sending connection context
0925 04:44:43.762758 (+ 191us) client_negotiation.cc:240] Negotiation successful
0925 04:44:43.762912 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":157,"threads_started":1}
I20260925 04:44:43.764394 7234 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.125" port: 45861 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 04:44:43 dist-test-slave-kghz krb5kdc[6471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311483, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.124@KRBTEST.COM for oryx/127.0.98.125@KRBTEST.COM
I20260925 04:44:43.773195 7237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:43.764972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:45861 (local address 127.0.0.1:49442)
0925 04:44:43.765087 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:43.765102 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:43.765179 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:43.765652 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:43.765654 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:43.765787 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:43.765891 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:43.765895 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:43.766952 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:43.766954 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:43.767527 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:43.767535 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:43.767961 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:43.767964 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:43.768036 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:43.768489 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:43.768500 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:43.769845 (+ 1345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:44:43.771734 (+ 1889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:43.771736 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:43.771738 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:43.771975 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:43.772300 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:43.772303 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:43.772304 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:43.772388 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:43.772781 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:44:43.772784 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:44:43.772864 (+ 80us) client_negotiation.cc:769] Sending connection context
0925 04:44:43.772958 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 04:44:43.773072 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 04:44:43.773295 7238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:43.765082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49442 (local address 127.0.98.125:45861)
0925 04:44:43.765375 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:43.765378 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:43.765391 (+ 13us) server_negotiation.cc:407] Connection header received
0925 04:44:43.765427 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:43.765429 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:43.765483 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:43.765566 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:43.766063 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:43.766835 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:43.767638 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:43.767866 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:43.767951 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:43.770027 (+ 2076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:43.770031 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:43.770033 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:43.770059 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:43.771633 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:43.772103 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:43.772106 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:43.772107 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:43.772188 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:43.772485 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:43.772487 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:43.772488 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:43.772693 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:43.772769 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:43.773015 (+ 246us) server_negotiation.cc:299] Negotiation successful
0925 04:44:43.773123 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":130,"threads_started":1}
I20260925 04:44:43.774065 7234 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.124" port: 41887 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 04:44:43 dist-test-slave-kghz krb5kdc[6471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311483, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.124@KRBTEST.COM for oryx/127.0.98.124@KRBTEST.COM
I20260925 04:44:43.782693 7237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:43.774553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:41887 (local address 127.0.0.1:45576)
0925 04:44:43.774663 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:43.774678 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:43.774750 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:43.774998 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:43.774999 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:43.775121 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:43.775187 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:43.775191 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:43.776360 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:43.776362 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:43.776910 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:43.776917 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:43.777363 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:43.777366 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:43.777457 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:43.777892 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:43.777902 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:43.779243 (+ 1341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:44:43.781000 (+ 1757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:43.781002 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:43.781004 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:43.781185 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:43.781518 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:43.781520 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:43.781522 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:43.781605 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:43.782085 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:44:43.782089 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:44:43.782190 (+ 101us) client_negotiation.cc:769] Sending connection context
0925 04:44:43.782392 (+ 202us) client_negotiation.cc:240] Negotiation successful
0925 04:44:43.782528 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":16}
I20260925 04:44:43.782712 7236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:43.774699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45576 (local address 127.0.98.124:41887)
0925 04:44:43.774837 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:43.774841 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:43.774853 (+ 12us) server_negotiation.cc:407] Connection header received
0925 04:44:43.774883 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:43.774885 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:43.774920 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:43.774994 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:43.775278 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:43.776223 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:43.777030 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:43.777266 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:43.777355 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:43.779379 (+ 2024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:43.779382 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:43.779384 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:43.779411 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:43.780895 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:43.781271 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:43.781273 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:43.781275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:43.781343 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:43.781744 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:43.781747 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:43.781748 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:43.781931 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:43.782027 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:43.782456 (+ 429us) server_negotiation.cc:299] Negotiation successful
0925 04:44:43.782582 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":64}
I20260925 04:44:43.785733 7234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b655e5719e73450cb120a45e38394900: Bootstrap starting.
I20260925 04:44:43.788322 7234 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b655e5719e73450cb120a45e38394900: Neither blocks nor log segments found. Creating new log.
I20260925 04:44:43.789165 7234 log.cc:824] T 00000000000000000000000000000000 P b655e5719e73450cb120a45e38394900: Log is configured to *not* fsync() on all Append() calls
I20260925 04:44:43.791301 7234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b655e5719e73450cb120a45e38394900: No bootstrap required, opened a new log
I20260925 04:44:43.794579 7234 raft_consensus.cc:348] T 00000000000000000000000000000000 P b655e5719e73450cb120a45e38394900 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58edb7db5329430685f0d9e7a5098394" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46181 } } peers { permanent_uuid: "4f99000937494cc99e7b9bd62901f3f0" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 45861 } } peers { permanent_uuid: "b655e5719e73450cb120a45e38394900" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 41887 } }
I20260925 04:44:43.794807 7234 raft_consensus.cc:374] T 00000000000000000000000000000000 P b655e5719e73450cb120a45e38394900 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:44:43.794889 7234 raft_consensus.cc:729] T 00000000000000000000000000000000 P b655e5719e73450cb120a45e38394900 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b655e5719e73450cb120a45e38394900, State: Initialized, Role: FOLLOWER
I20260925 04:44:43.795397 7234 consensus_queue.cc:260] T 00000000000000000000000000000000 P b655e5719e73450cb120a45e38394900 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58edb7db5329430685f0d9e7a5098394" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46181 } } peers { permanent_uuid: "4f99000937494cc99e7b9bd62901f3f0" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 45861 } } peers { permanent_uuid: "b655e5719e73450cb120a45e38394900" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 41887 } }
I20260925 04:44:43.796383 7240 sys_catalog.cc:455] T 00000000000000000000000000000000 P b655e5719e73450cb120a45e38394900 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58edb7db5329430685f0d9e7a5098394" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46181 } } peers { permanent_uuid: "4f99000937494cc99e7b9bd62901f3f0" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 45861 } } peers { permanent_uuid: "b655e5719e73450cb120a45e38394900" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 41887 } } }
I20260925 04:44:43.796557 7240 sys_catalog.cc:458] T 00000000000000000000000000000000 P b655e5719e73450cb120a45e38394900 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:44:43.796665 7234 sys_catalog.cc:564] T 00000000000000000000000000000000 P b655e5719e73450cb120a45e38394900 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 04:44:43.801806 7251 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b655e5719e73450cb120a45e38394900: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 04:44:43.801903 7251 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 04:44:43.805985 7082 master_runner.cc:428] Detected that this master b655e5719e73450cb120a45e38394900 is joining an existing cluster
I20260925 04:44:43.806070 7082 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.124:41887
I20260925 04:44:43.812464 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 6578
I20260925 04:44:43.819509 7259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:43.808776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:46181 (local address 127.0.0.1:39376)
0925 04:44:43.809266 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:43.809281 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:43.809375 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:43.809759 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:43.809764 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:43.810016 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:43.810175 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:43.810183 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:43.811438 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:43.811442 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:43.812500 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:43.812511 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:43.814395 (+ 1884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:43.814398 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:43.814512 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:43.814985 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:43.815001 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:43.815549 (+ 548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:44:43.817700 (+ 2151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:43.817704 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:43.817707 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:43.817974 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:43.818313 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:43.818317 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:43.818319 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:43.818416 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:43.818940 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:44:43.818943 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:44:43.819039 (+ 96us) client_negotiation.cc:769] Sending connection context
0925 04:44:43.819150 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 04:44:43.819297 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"mutex_wait_us":1,"thread_start_us":118,"threads_started":1}
I20260925 04:44:43.819896 6555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:43.809261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39376 (local address 127.0.98.126:46181)
0925 04:44:43.809397 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:43.809401 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:43.809418 (+ 17us) server_negotiation.cc:407] Connection header received
0925 04:44:43.809468 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:43.809472 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:43.809536 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:43.809626 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:43.810364 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:43.811282 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:43.813776 (+ 2494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:43.814125 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:43.814203 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:43.815724 (+ 1521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:43.815728 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:43.815732 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:43.815761 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:43.817523 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:43.818081 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:43.818085 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:43.818087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:43.818173 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:43.818552 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:43.818556 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:43.818558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:43.818790 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:43.818866 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:43.819242 (+ 376us) server_negotiation.cc:299] Negotiation successful
0925 04:44:43.819373 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 04:44:43.821040 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:46321
--local_ip_for_outbound_sockets=127.0.98.65
--tserver_master_addrs=127.0.98.126:46181
--webserver_port=46493
--webserver_interface=127.0.98.65
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/krb5kdc/oryx_127.0.98.65.keytab
--principal=oryx/127.0.98.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:33935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.126:46181,127.0.98.125:45861
--tserver_master_addrs=127.0.98.126:46181,127.0.98.125:45861,127.0.98.124:41887 with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 04:44:43.821933 7238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:43.808822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49444 (local address 127.0.98.125:45861)
0925 04:44:43.809338 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:43.809344 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:43.809766 (+ 422us) server_negotiation.cc:407] Connection header received
0925 04:44:43.809913 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:43.809915 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:43.809962 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:43.810013 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:43.810783 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:43.811542 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:43.815257 (+ 3715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:43.815640 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:43.815728 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:43.817976 (+ 2248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:43.817979 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:43.817982 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:43.818010 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:43.819730 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:43.820694 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:43.820699 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:43.820701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:43.820785 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:43.821134 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:43.821139 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:43.821140 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:43.821314 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:43.821418 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:43.821700 (+ 282us) server_negotiation.cc:299] Negotiation successful
0925 04:44:43.821821 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424}
I20260925 04:44:43.822688 7236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:43.809207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45578 (local address 127.0.98.124:41887)
0925 04:44:43.809420 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:43.809427 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:43.809656 (+ 229us) server_negotiation.cc:407] Connection header received
0925 04:44:43.809798 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:43.809801 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:43.809837 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:43.809889 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:43.810766 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:43.811542 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:43.813113 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:43.813462 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:43.813549 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:43.815455 (+ 1906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:43.815459 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:43.815461 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:43.815481 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:43.817691 (+ 2210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:43.819873 (+ 2182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:43.819877 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:43.819879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:43.819962 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:43.820280 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:43.820286 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:43.820291 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:43.820563 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:43.820660 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:43.822412 (+ 1752us) server_negotiation.cc:299] Negotiation successful
0925 04:44:43.822524 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153}
I20260925 04:44:43.822714 7257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:43.808964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:45861 (local address 127.0.0.1:49444)
0925 04:44:43.809696 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:43.809709 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:43.809815 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:43.810144 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:43.810150 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:43.810464 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:43.810648 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:43.810656 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:43.811691 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:43.811694 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:43.815115 (+ 3421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:43.815125 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:43.815888 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:43.815892 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:43.816082 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:43.816516 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:43.816537 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:43.817819 (+ 1282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:44:43.820299 (+ 2480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:43.820302 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:43.820304 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:43.820573 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:43.820969 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:43.820971 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:43.820972 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:43.821039 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:43.821407 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:44:43.821410 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:44:43.821527 (+ 117us) client_negotiation.cc:769] Sending connection context
0925 04:44:43.822085 (+ 558us) client_negotiation.cc:240] Negotiation successful
0925 04:44:43.822232 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"mutex_wait_us":24,"spinlock_wait_cycles":4992,"thread_start_us":81,"threads_started":1}
I20260925 04:44:43.823364 7258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:43.808879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:41887 (local address 127.0.0.1:45578)
0925 04:44:43.809585 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:43.809597 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:43.809704 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:43.810102 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:43.810106 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:43.810447 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:43.810649 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:43.810656 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:43.811842 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:43.811844 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:43.812991 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:43.813004 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:43.813701 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:43.813707 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:43.813827 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:43.814478 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:43.814493 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:43.815349 (+ 856us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:44:43.817817 (+ 2468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:43.817821 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:43.817823 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:43.818972 (+ 1149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:43.820069 (+ 1097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:43.820072 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:43.820074 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:43.820181 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:43.820703 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:44:43.820706 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:44:43.820806 (+ 100us) client_negotiation.cc:769] Sending connection context
0925 04:44:43.823075 (+ 2269us) client_negotiation.cc:240] Negotiation successful
0925 04:44:43.823202 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"mutex_wait_us":54,"thread_start_us":98,"threads_started":1}
Sep 25 04:44:43 dist-test-slave-kghz krb5kdc[6471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311482, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.126@KRBTEST.COM for oryx/127.0.98.124@KRBTEST.COM
I20260925 04:44:43.839941 7240 raft_consensus.cc:482] T 00000000000000000000000000000000 P b655e5719e73450cb120a45e38394900 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:44:43.840169 7240 raft_consensus.cc:504] T 00000000000000000000000000000000 P b655e5719e73450cb120a45e38394900 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58edb7db5329430685f0d9e7a5098394" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46181 } } peers { permanent_uuid: "4f99000937494cc99e7b9bd62901f3f0" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 45861 } } peers { permanent_uuid: "b655e5719e73450cb120a45e38394900" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 41887 } }
I20260925 04:44:43.841208 7240 leader_election.cc:288] T 00000000000000000000000000000000 P b655e5719e73450cb120a45e38394900 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58edb7db5329430685f0d9e7a5098394 (127.0.98.126:46181), 4f99000937494cc99e7b9bd62901f3f0 (127.0.98.125:45861)
I20260925 04:44:43.841691 6526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b655e5719e73450cb120a45e38394900" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58edb7db5329430685f0d9e7a5098394" is_pre_election: true
I20260925 04:44:43.841743 7126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b655e5719e73450cb120a45e38394900" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f99000937494cc99e7b9bd62901f3f0" is_pre_election: true
I20260925 04:44:43.841893 7126 raft_consensus.cc:1799] T 00000000000000000000000000000000 P 4f99000937494cc99e7b9bd62901f3f0 [term 1 LEARNER]: Handling vote request from an unknown peer b655e5719e73450cb120a45e38394900
I20260925 04:44:43.842016 7126 raft_consensus.cc:2396] T 00000000000000000000000000000000 P 4f99000937494cc99e7b9bd62901f3f0 [term 1 LEARNER]: Leader pre-election vote request: Denying vote to candidate b655e5719e73450cb120a45e38394900 for term 1 because replica has last-logged OpId of term: 1 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260925 04:44:43.842494 7182 leader_election.cc:302] T 00000000000000000000000000000000 P b655e5719e73450cb120a45e38394900 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b655e5719e73450cb120a45e38394900; no voters: 4f99000937494cc99e7b9bd62901f3f0, 58edb7db5329430685f0d9e7a5098394
I20260925 04:44:43.842964 7240 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b655e5719e73450cb120a45e38394900 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:44:43.860587 7240 raft_consensus.cc:2726] T 00000000000000000000000000000000 P b655e5719e73450cb120a45e38394900 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 04:44:43.862469 7236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:43.825409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45584 (local address 127.0.98.124:41887)
0925 04:44:43.825525 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:43.825528 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:43.825540 (+ 12us) server_negotiation.cc:407] Connection header received
0925 04:44:43.825589 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:43.825591 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:43.825630 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:43.825696 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:43.826236 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:43.826782 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:43.827691 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:43.827917 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:43.828035 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:43.830345 (+ 2310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:43.830348 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:43.830351 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:43.830374 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:43.860450 (+ 30076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:43.861113 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:43.861118 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:43.861120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:43.861240 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:43.861555 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:43.861560 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:43.861562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:43.861760 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:43.861871 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:43.862153 (+ 282us) server_negotiation.cc:299] Negotiation successful
0925 04:44:43.862303 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 04:44:43.862469 7073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:43.824919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:41887 (local address 127.0.0.1:45584)
0925 04:44:43.825176 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:43.825191 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:43.825272 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:43.825772 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:43.825776 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:43.825990 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:43.826100 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:43.826108 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:43.826899 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:43.826905 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:43.827576 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:43.827586 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:43.828037 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:43.828041 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:43.828131 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:43.828760 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:43.828781 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:43.830119 (+ 1338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:44:43.860647 (+ 30528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:43.860651 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:43.860656 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:43.860989 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:43.861350 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:43.861353 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:43.861355 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:43.861455 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:43.861860 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:44:43.861863 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:44:43.861972 (+ 109us) client_negotiation.cc:769] Sending connection context
0925 04:44:43.862090 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 04:44:43.862303 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177}
I20260925 04:44:43.863899 6516 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.124:41887: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "b655e5719e73450cb120a45e38394900"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.124"
port: 41887
}
attrs {
promote: true
}
}
dest_uuid: "58edb7db5329430685f0d9e7a5098394"
cas_config_opid_index: 5
I20260925 04:44:43.864631 6516 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58edb7db5329430685f0d9e7a5098394 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "58edb7db5329430685f0d9e7a5098394" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46181 } } peers { permanent_uuid: "4f99000937494cc99e7b9bd62901f3f0" member_type: NON_VOTER last_known_addr { host: "127.0.98.125" port: 45861 } attrs { promote: true } } peers { permanent_uuid: "b655e5719e73450cb120a45e38394900" member_type: NON_VOTER last_known_addr { host: "127.0.98.124" port: 41887 } attrs { promote: true } }
I20260925 04:44:43.865768 7075 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 58edb7db5329430685f0d9e7a5098394 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER b655e5719e73450cb120a45e38394900 (127.0.98.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "58edb7db5329430685f0d9e7a5098394" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46181 } } peers { permanent_uuid: "4f99000937494cc99e7b9bd62901f3f0" member_type: NON_VOTER last_known_addr { host: "127.0.98.125" port: 45861 } attrs { promote: true } } peers { permanent_uuid: "b655e5719e73450cb120a45e38394900" member_type: NON_VOTER last_known_addr { host: "127.0.98.124" port: 41887 } attrs { promote: true } } }
I20260925 04:44:43.867056 7075 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.124:41887
I20260925 04:44:43.867230 7075 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58edb7db5329430685f0d9e7a5098394 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "58edb7db5329430685f0d9e7a5098394" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "58edb7db5329430685f0d9e7a5098394" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46181 } } peers { permanent_uuid: "4f99000937494cc99e7b9bd62901f3f0" member_type: NON_VOTER last_known_addr { host: "127.0.98.125" port: 45861 } attrs { promote: true } } peers { permanent_uuid: "b655e5719e73450cb120a45e38394900" member_type: NON_VOTER last_known_addr { host: "127.0.98.124" port: 41887 } attrs { promote: true } } }
I20260925 04:44:43.867349 7075 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58edb7db5329430685f0d9e7a5098394 [sys.catalog]: This master's current role is: LEADER
I20260925 04:44:43.868064 7082 master.cc:537] Master@127.0.98.124:41887 shutting down...
W20260925 04:44:43.875574 6503 consensus_peers.cc:487] T 00000000000000000000000000000000 P 58edb7db5329430685f0d9e7a5098394 -> Peer b655e5719e73450cb120a45e38394900 (127.0.98.124:41887): Couldn't send request to peer b655e5719e73450cb120a45e38394900. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 04:44:43.878818 7082 raft_consensus.cc:2229] T 00000000000000000000000000000000 P b655e5719e73450cb120a45e38394900 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:44:43.879128 7082 raft_consensus.cc:2258] T 00000000000000000000000000000000 P b655e5719e73450cb120a45e38394900 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:44:43.879213 7082 tablet_replica.cc:330] T 00000000000000000000000000000000 P b655e5719e73450cb120a45e38394900: stopping tablet replica
I20260925 04:44:43.883397 7073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:43.865935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:45861 (local address 127.0.0.1:49450)
0925 04:44:43.877270 (+ 11335us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:43.877289 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:43.877410 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:43.878327 (+ 917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:43.878330 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:43.878350 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:44:43.878465 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:43.878477 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:43.879847 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:43.879850 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:43.881230 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:43.881243 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:43.882835 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:43.882842 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:43.882963 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:43.883036 (+ 73us) client_negotiation.cc:769] Sending connection context
0925 04:44:43.883114 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 04:44:43.883213 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11229}
I20260925 04:44:43.883944 7238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:43.867591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49450 (local address 127.0.98.125:45861)
0925 04:44:43.867748 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:43.867755 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:43.877531 (+ 9776us) server_negotiation.cc:407] Connection header received
0925 04:44:43.877584 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:43.877586 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:43.877637 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:43.877738 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:44:43.878629 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:43.879388 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:43.881395 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:43.882464 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:43.882557 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:43.882727 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:43.883713 (+ 986us) server_negotiation.cc:299] Negotiation successful
0925 04:44:43.883813 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 04:44:43.884295 7082 master.cc:552] Master@127.0.98.124:41887 shutdown complete.
I20260925 04:44:43.884346 7082 master_runner.cc:305] Clearing existing system tablet
I20260925 04:44:43.884629 7082 fs_manager.cc:359] Metadata directory not provided
I20260925 04:44:43.884689 7082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:44:43.884871 7126 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4f99000937494cc99e7b9bd62901f3f0 [term 1 LEARNER]: Refusing update from remote peer 58edb7db5329430685f0d9e7a5098394: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 04:44:43.885398 7076 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 58edb7db5329430685f0d9e7a5098394 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f99000937494cc99e7b9bd62901f3f0" member_type: NON_VOTER last_known_addr { host: "127.0.98.125" port: 45861 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260925 04:44:43.886780 7082 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20260925 04:44:43.887588 7126 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4f99000937494cc99e7b9bd62901f3f0 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER b655e5719e73450cb120a45e38394900 (127.0.98.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "58edb7db5329430685f0d9e7a5098394" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46181 } } peers { permanent_uuid: "4f99000937494cc99e7b9bd62901f3f0" member_type: NON_VOTER last_known_addr { host: "127.0.98.125" port: 45861 } attrs { promote: true } } peers { permanent_uuid: "b655e5719e73450cb120a45e38394900" member_type: NON_VOTER last_known_addr { host: "127.0.98.124" port: 41887 } attrs { promote: true } } }
I20260925 04:44:43.889382 7155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f99000937494cc99e7b9bd62901f3f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58edb7db5329430685f0d9e7a5098394. Latest consensus state: current_term: 1 leader_uuid: "58edb7db5329430685f0d9e7a5098394" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "58edb7db5329430685f0d9e7a5098394" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46181 } } peers { permanent_uuid: "4f99000937494cc99e7b9bd62901f3f0" member_type: NON_VOTER last_known_addr { host: "127.0.98.125" port: 45861 } attrs { promote: true } } peers { permanent_uuid: "b655e5719e73450cb120a45e38394900" member_type: NON_VOTER last_known_addr { host: "127.0.98.124" port: 41887 } attrs { promote: true } } }
I20260925 04:44:43.889497 7155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f99000937494cc99e7b9bd62901f3f0 [sys.catalog]: This master's current role is: LEARNER
I20260925 04:44:43.889519 7264 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f99000937494cc99e7b9bd62901f3f0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "58edb7db5329430685f0d9e7a5098394" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "58edb7db5329430685f0d9e7a5098394" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46181 } } peers { permanent_uuid: "4f99000937494cc99e7b9bd62901f3f0" member_type: NON_VOTER last_known_addr { host: "127.0.98.125" port: 45861 } attrs { promote: true } } peers { permanent_uuid: "b655e5719e73450cb120a45e38394900" member_type: NON_VOTER last_known_addr { host: "127.0.98.124" port: 41887 } attrs { promote: true } } }
I20260925 04:44:43.889611 7264 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f99000937494cc99e7b9bd62901f3f0 [sys.catalog]: This master's current role is: LEARNER
I20260925 04:44:43.889989 7263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:43.890285 7082 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:44:43.890411 7082 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260925 04:44:43.890450 7082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/master-2/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/master-2/wal
uuid: "b655e5719e73450cb120a45e38394900"
format_stamp: "Formatted at 2026-09-25 04:44:43 on dist-test-slave-kghz"
server_key: "d65be6603cd4045170caf0c5b676a0b2"
server_key_iv: "68735a17b89bc191cebba0866a05a7ec"
server_key_version: "encryptionkey@0"
I20260925 04:44:43.890475 7076 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 58edb7db5329430685f0d9e7a5098394: attempting to promote NON_VOTER 4f99000937494cc99e7b9bd62901f3f0 to VOTER
I20260925 04:44:43.890933 7076 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58edb7db5329430685f0d9e7a5098394 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "58edb7db5329430685f0d9e7a5098394" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46181 } } peers { permanent_uuid: "4f99000937494cc99e7b9bd62901f3f0" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 45861 } attrs { promote: false } } peers { permanent_uuid: "b655e5719e73450cb120a45e38394900" member_type: NON_VOTER last_known_addr { host: "127.0.98.124" port: 41887 } attrs { promote: true } }
I20260925 04:44:43.892583 7082 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P b655e5719e73450cb120a45e38394900: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:44:43.892789 7126 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4f99000937494cc99e7b9bd62901f3f0 [term 1 LEARNER]: Refusing update from remote peer 58edb7db5329430685f0d9e7a5098394: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260925 04:44:43.893193 7075 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 58edb7db5329430685f0d9e7a5098394 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f99000937494cc99e7b9bd62901f3f0" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 45861 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260925 04:44:43.894472 7073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:43.892008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:41887 (local address 127.0.0.1:45586)
0925 04:44:43.894268 (+ 2260us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:43.894363 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:41887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2146}
I20260925 04:44:43.895354 7073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:43.895056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:41887 (local address 127.0.0.1:45600)
0925 04:44:43.895239 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:43.895289 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:41887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
W20260925 04:44:43.895565 6503 consensus_peers.cc:487] T 00000000000000000000000000000000 P 58edb7db5329430685f0d9e7a5098394 -> Peer b655e5719e73450cb120a45e38394900 (127.0.98.124:41887): Couldn't send request to peer b655e5719e73450cb120a45e38394900. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.124:41887: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 04:44:43.895475 7075 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 58edb7db5329430685f0d9e7a5098394 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 4f99000937494cc99e7b9bd62901f3f0 (127.0.98.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "58edb7db5329430685f0d9e7a5098394" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46181 } } peers { permanent_uuid: "4f99000937494cc99e7b9bd62901f3f0" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 45861 } attrs { promote: false } } peers { permanent_uuid: "b655e5719e73450cb120a45e38394900" member_type: NON_VOTER last_known_addr { host: "127.0.98.124" port: 41887 } attrs { promote: true } } }
I20260925 04:44:43.896662 7082 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P b655e5719e73450cb120a45e38394900: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 04:44:43.896842 7082 log.cc:1196] T 00000000000000000000000000000000 P b655e5719e73450cb120a45e38394900: Deleting WAL directory at /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260925 04:44:43.896946 7075 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58edb7db5329430685f0d9e7a5098394 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "58edb7db5329430685f0d9e7a5098394" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "58edb7db5329430685f0d9e7a5098394" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46181 } } peers { permanent_uuid: "4f99000937494cc99e7b9bd62901f3f0" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 45861 } attrs { promote: false } } peers { permanent_uuid: "b655e5719e73450cb120a45e38394900" member_type: NON_VOTER last_known_addr { host: "127.0.98.124" port: 41887 } attrs { promote: true } } }
I20260925 04:44:43.897068 7075 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58edb7db5329430685f0d9e7a5098394 [sys.catalog]: This master's current role is: LEADER
I20260925 04:44:43.897217 7075 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58edb7db5329430685f0d9e7a5098394 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "58edb7db5329430685f0d9e7a5098394" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "58edb7db5329430685f0d9e7a5098394" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46181 } } peers { permanent_uuid: "4f99000937494cc99e7b9bd62901f3f0" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 45861 } attrs { promote: false } } peers { permanent_uuid: "b655e5719e73450cb120a45e38394900" member_type: NON_VOTER last_known_addr { host: "127.0.98.124" port: 41887 } attrs { promote: true } } }
I20260925 04:44:43.897323 7075 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58edb7db5329430685f0d9e7a5098394 [sys.catalog]: This master's current role is: LEADER
I20260925 04:44:43.897600 7082 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P b655e5719e73450cb120a45e38394900: Deleting consensus metadata
I20260925 04:44:43.897940 7082 master_runner.cc:315] Copying system tablet from 127.0.98.126:46181
I20260925 04:44:43.898118 7126 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4f99000937494cc99e7b9bd62901f3f0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 4f99000937494cc99e7b9bd62901f3f0 (127.0.98.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "58edb7db5329430685f0d9e7a5098394" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46181 } } peers { permanent_uuid: "4f99000937494cc99e7b9bd62901f3f0" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 45861 } attrs { promote: false } } peers { permanent_uuid: "b655e5719e73450cb120a45e38394900" member_type: NON_VOTER last_known_addr { host: "127.0.98.124" port: 41887 } attrs { promote: true } } }
I20260925 04:44:43.899366 7082 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P b655e5719e73450cb120a45e38394900: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.126:46181
I20260925 04:44:43.900574 7264 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f99000937494cc99e7b9bd62901f3f0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "58edb7db5329430685f0d9e7a5098394" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "58edb7db5329430685f0d9e7a5098394" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46181 } } peers { permanent_uuid: "4f99000937494cc99e7b9bd62901f3f0" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 45861 } attrs { promote: false } } peers { permanent_uuid: "b655e5719e73450cb120a45e38394900" member_type: NON_VOTER last_known_addr { host: "127.0.98.124" port: 41887 } attrs { promote: true } } }
I20260925 04:44:43.900692 7264 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f99000937494cc99e7b9bd62901f3f0 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:44:43.908438 7271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:43.900510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:46181 (local address 127.0.0.1:39392)
0925 04:44:43.900901 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:43.900921 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:43.901057 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:43.901314 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:43.901317 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:43.901528 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:43.901631 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:43.901638 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:43.902533 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:43.902538 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:43.903147 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:43.903156 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:43.903633 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:43.903636 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:43.903737 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:43.904354 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:43.904367 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:43.904789 (+ 422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:44:43.906758 (+ 1969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:43.906763 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:43.906765 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:43.906986 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:43.907344 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:43.907348 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:43.907350 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:43.907436 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:43.907841 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:44:43.907844 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:44:43.907959 (+ 115us) client_negotiation.cc:769] Sending connection context
0925 04:44:43.908066 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 04:44:43.908228 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":63,"threads_started":1}
I20260925 04:44:43.908424 6555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:43.900302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39392 (local address 127.0.98.126:46181)
0925 04:44:43.900408 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:43.900412 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:43.901010 (+ 598us) server_negotiation.cc:407] Connection header received
0925 04:44:43.901152 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:43.901155 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:43.901207 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:43.901297 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:43.901753 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:43.902425 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:43.903276 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:43.903518 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:43.903629 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:43.904929 (+ 1300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:43.904933 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:43.904935 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:43.904960 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:43.906645 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:43.907088 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:43.907093 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:43.907095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:43.907208 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:43.907539 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:43.907543 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:43.907545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:43.907740 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:43.907853 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:43.908117 (+ 264us) server_negotiation.cc:299] Negotiation successful
0925 04:44:43.908228 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 04:44:43.909242 6536 tablet_copy_service.cc:140] P 58edb7db5329430685f0d9e7a5098394: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer b655e5719e73450cb120a45e38394900 ({username='oryx', principal='oryx/127.0.98.124@KRBTEST.COM'} at 127.0.0.1:39392)
I20260925 04:44:43.909363 6536 tablet_copy_service.cc:161] P 58edb7db5329430685f0d9e7a5098394: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer b655e5719e73450cb120a45e38394900 at {username='oryx', principal='oryx/127.0.98.124@KRBTEST.COM'} at 127.0.0.1:39392: session id = b655e5719e73450cb120a45e38394900-00000000000000000000000000000000
I20260925 04:44:43.910303 6536 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 58edb7db5329430685f0d9e7a5098394: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 04:44:43.911585 7082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 04:44:43.913558 7082 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P b655e5719e73450cb120a45e38394900: tablet copy: Starting download of 0 data blocks...
I20260925 04:44:43.913759 7082 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P b655e5719e73450cb120a45e38394900: tablet copy: Starting download of 1 WAL segments...
I20260925 04:44:43.915279 7082 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P b655e5719e73450cb120a45e38394900: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 04:44:43.916970 6536 tablet_copy_service.cc:342] P 58edb7db5329430685f0d9e7a5098394: Request end of tablet copy session b655e5719e73450cb120a45e38394900-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.124@KRBTEST.COM'} at 127.0.0.1:39392
I20260925 04:44:43.917071 6536 tablet_copy_service.cc:433] P 58edb7db5329430685f0d9e7a5098394: ending tablet copy session b655e5719e73450cb120a45e38394900-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer b655e5719e73450cb120a45e38394900
I20260925 04:44:43.924136 7082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:44:43.926020 7272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:43.926445 7275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:43.926632 7082 server_base.cc:1034] running on GCE node
W20260925 04:44:43.926620 7273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:43.926904 7082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:44:43.927249 7082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:44:43.928392 7082 hybrid_clock.cc:648] HybridClock initialized: now 1790311483928389 us; error 26 us; skew 500 ppm
Sep 25 04:44:43 dist-test-slave-kghz krb5kdc[6471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311483, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:44:43.930032 7082 init.cc:377] Logged in from keytab as oryx/127.0.98.124@KRBTEST.COM (short username oryx)
I20260925 04:44:43.930585 7082 webserver.cc:460] Webserver started at http://127.0.98.124:46493/ using document root <none> and password file <none>
I20260925 04:44:43.930783 7082 fs_manager.cc:359] Metadata directory not provided
I20260925 04:44:43.930841 7082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:44:43.932003 7082 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 04:44:43.932965 7282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:43.933200 7082 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 04:44:43.933291 7082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/master-2/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/master-2/wal
uuid: "b655e5719e73450cb120a45e38394900"
format_stamp: "Formatted at 2026-09-25 04:44:43 on dist-test-slave-kghz"
server_key: "d65be6603cd4045170caf0c5b676a0b2"
server_key_iv: "68735a17b89bc191cebba0866a05a7ec"
server_key_version: "encryptionkey@0"
I20260925 04:44:43.933377 7082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:44:43.966449 7082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:44:43.968629 7082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:44:43.973809 7082 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.124:41887
I20260925 04:44:43.973887 6841 heartbeater.cc:503] Master 127.0.98.126:46181 was elected leader, sending a full tablet report...
I20260925 04:44:43.974061 7334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.124:41887 every 8 connection(s)
I20260925 04:44:43.974514 7082 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/master-2/data/info.pb
I20260925 04:44:43.976958 7335 sys_catalog.cc:263] Verifying existing consensus state
I20260925 04:44:43.978029 7335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b655e5719e73450cb120a45e38394900: Bootstrap starting.
I20260925 04:44:43.990084 7335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b655e5719e73450cb120a45e38394900: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 04:44:43.990461 7335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b655e5719e73450cb120a45e38394900: Bootstrap complete.
I20260925 04:44:43.991073 7335 raft_consensus.cc:348] T 00000000000000000000000000000000 P b655e5719e73450cb120a45e38394900 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "58edb7db5329430685f0d9e7a5098394" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46181 } } peers { permanent_uuid: "4f99000937494cc99e7b9bd62901f3f0" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 45861 } attrs { promote: false } } peers { permanent_uuid: "b655e5719e73450cb120a45e38394900" member_type: NON_VOTER last_known_addr { host: "127.0.98.124" port: 41887 } attrs { promote: true } }
I20260925 04:44:43.991219 7335 raft_consensus.cc:729] T 00000000000000000000000000000000 P b655e5719e73450cb120a45e38394900 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b655e5719e73450cb120a45e38394900, State: Initialized, Role: LEARNER
I20260925 04:44:43.991319 7335 consensus_queue.cc:260] T 00000000000000000000000000000000 P b655e5719e73450cb120a45e38394900 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "58edb7db5329430685f0d9e7a5098394" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46181 } } peers { permanent_uuid: "4f99000937494cc99e7b9bd62901f3f0" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 45861 } attrs { promote: false } } peers { permanent_uuid: "b655e5719e73450cb120a45e38394900" member_type: NON_VOTER last_known_addr { host: "127.0.98.124" port: 41887 } attrs { promote: true } }
I20260925 04:44:43.991667 7341 sys_catalog.cc:455] T 00000000000000000000000000000000 P b655e5719e73450cb120a45e38394900 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "58edb7db5329430685f0d9e7a5098394" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46181 } } peers { permanent_uuid: "4f99000937494cc99e7b9bd62901f3f0" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 45861 } attrs { promote: false } } peers { permanent_uuid: "b655e5719e73450cb120a45e38394900" member_type: NON_VOTER last_known_addr { host: "127.0.98.124" port: 41887 } attrs { promote: true } } }
I20260925 04:44:43.991773 7335 sys_catalog.cc:564] T 00000000000000000000000000000000 P b655e5719e73450cb120a45e38394900 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:44:43.991886 7341 sys_catalog.cc:458] T 00000000000000000000000000000000 P b655e5719e73450cb120a45e38394900 [sys.catalog]: This master's current role is: LEARNER
I20260925 04:44:43.995678 7352 catalog_manager.cc:1259] Loaded cluster ID: 279c26e79db84d6982461caf8787f885
I20260925 04:44:43.995832 7352 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b655e5719e73450cb120a45e38394900: loading cluster ID for follower catalog manager: success
W20260925 04:44:43.997373 7260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:44:43.997663 7260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:44:43.997748 7260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260925 04:44:43.998070 7352 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b655e5719e73450cb120a45e38394900: acquiring CA information for follower catalog manager: success
I20260925 04:44:43.999574 7352 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b655e5719e73450cb120a45e38394900: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 04:44:44.002381 7260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:44:44.002475 7260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:44:44.002576 7260 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260925 04:44:44.008282 7260 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:33935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/krb5kdc/oryx_127.0.98.65.keytab
--principal=oryx/127.0.98.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:46321
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=46493
--tserver_master_addrs=127.0.98.126:46181,127.0.98.125:45861,127.0.98.124:41887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7260
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:44:44.009404 7260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:44:44.010576 7260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:44:44.021575 7354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:44.021569 7355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:44.021837 7357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:44.022311 7260 server_base.cc:1034] running on GCE node
I20260925 04:44:44.022738 7260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:44:44.023427 7260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:44:44.024613 7260 hybrid_clock.cc:648] HybridClock initialized: now 1790311484024594 us; error 37 us; skew 500 ppm
Sep 25 04:44:44 dist-test-slave-kghz krb5kdc[6471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311484, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:44:44.027837 7260 init.cc:377] Logged in from keytab as oryx/127.0.98.65@KRBTEST.COM (short username oryx)
I20260925 04:44:44.028916 7260 webserver.cc:460] Webserver started at http://127.0.98.65:46493/ using document root <none> and password file <none>
I20260925 04:44:44.029506 7260 fs_manager.cc:359] Metadata directory not provided
I20260925 04:44:44.029582 7260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:44:44.033725 7260 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 04:44:44.036020 7364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:44.037312 7260 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 04:44:44.037425 7260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-0/wal
uuid: "0bbea834c4b244998ef191efd5bb4173"
format_stamp: "Formatted at 2026-09-25 04:44:42 on dist-test-slave-kghz"
server_key: "33630ecea24d8515f015f5e7e75a4179"
server_key_iv: "c11bf6845888a623dc8db0a6431aaea6"
server_key_version: "encryptionkey@0"
I20260925 04:44:44.038031 7260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:44:44.047993 7260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:44:44.051380 7260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:44:44.051597 7260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:44:44.052197 7260 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:44:44.053057 7260 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:44:44.053103 7260 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:44.053166 7260 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:44:44.053198 7260 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:44.062469 7260 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:46321
I20260925 04:44:44.062489 7477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:46321 every 8 connection(s)
I20260925 04:44:44.063725 7260 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
I20260925 04:44:44.067471 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 7260
I20260925 04:44:44.067664 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 6712
Sep 25 04:44:44 dist-test-slave-kghz krb5kdc[6471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311484, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM
I20260925 04:44:44.076823 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:36667
--local_ip_for_outbound_sockets=127.0.98.66
--tserver_master_addrs=127.0.98.126:46181
--webserver_port=41511
--webserver_interface=127.0.98.66
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/krb5kdc/oryx_127.0.98.66.keytab
--principal=oryx/127.0.98.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:33935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.126:46181,127.0.98.125:45861
--tserver_master_addrs=127.0.98.126:46181,127.0.98.125:45861,127.0.98.124:41887 with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 04:44:44 dist-test-slave-kghz krb5kdc[6471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311484, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.125@KRBTEST.COM
Sep 25 04:44:44 dist-test-slave-kghz krb5kdc[6471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311484, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.124@KRBTEST.COM
I20260925 04:44:44.083060 6555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:44.065760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:50061 (local address 127.0.98.126:46181)
0925 04:44:44.066017 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:44.066023 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:44.066977 (+ 954us) server_negotiation.cc:407] Connection header received
0925 04:44:44.068695 (+ 1718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:44.068698 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:44.068750 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:44.068869 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:44.070420 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:44.071355 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:44.073352 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:44.073689 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:44.073763 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:44.078629 (+ 4866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:44.078633 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:44.078637 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:44.078669 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:44.081115 (+ 2446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:44.081585 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:44.081588 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:44.081590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:44.081680 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:44.081955 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:44.081957 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:44.081959 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:44.082173 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:44.082268 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:44.082763 (+ 495us) server_negotiation.cc:299] Negotiation successful
0925 04:44:44.082888 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169}
I20260925 04:44:44.083060 7483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:44.066528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:45605 (local address 127.0.98.124:41887)
0925 04:44:44.068905 (+ 2377us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:44.068912 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:44.071960 (+ 3048us) server_negotiation.cc:407] Connection header received
0925 04:44:44.072145 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:44.072148 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:44.072199 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:44.072272 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:44.072907 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:44.073903 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:44.074992 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:44.075446 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:44.075567 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:44.078410 (+ 2843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:44.078414 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:44.078416 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:44.078443 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:44.080450 (+ 2007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:44.081204 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:44.081209 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:44.081211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:44.081303 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:44.081712 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:44.081718 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:44.081720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:44.081966 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:44.082047 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:44.082763 (+ 716us) server_negotiation.cc:299] Negotiation successful
0925 04:44:44.082889 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2276,"thread_start_us":86,"threads_started":1}
I20260925 04:44:44.083166 7238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:44.065882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:39537 (local address 127.0.98.125:45861)
0925 04:44:44.066407 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:44.066412 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:44.069174 (+ 2762us) server_negotiation.cc:407] Connection header received
0925 04:44:44.069274 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:44.069278 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:44.069333 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:44.069423 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:44.070805 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:44.071447 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:44.073375 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:44.073808 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:44.073901 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:44.078673 (+ 4772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:44.078680 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:44.078685 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:44.078720 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:44.080579 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:44.081236 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:44.081241 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:44.081245 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:44.081344 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:44.081736 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:44.081741 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:44.081744 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:44.082013 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:44.082107 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:44.082780 (+ 673us) server_negotiation.cc:299] Negotiation successful
0925 04:44:44.082922 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437}
I20260925 04:44:44.084802 7484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:44.066141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:41887 (local address 127.0.98.65:45605)
0925 04:44:44.071813 (+ 5672us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:44.071836 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:44.072033 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:44.072431 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:44.072434 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:44.072678 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:44.072775 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:44.072781 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:44.074023 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:44.074025 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:44.074873 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:44.074886 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:44.075597 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:44.075601 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:44.075690 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:44.076167 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:44.076181 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:44.078189 (+ 2008us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:44:44.080603 (+ 2414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:44.080611 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:44.080633 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:44.081082 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:44.081415 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:44.081419 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:44.081422 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:44.081608 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:44.082110 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:44:44.082116 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:44:44.082477 (+ 361us) client_negotiation.cc:769] Sending connection context
0925 04:44:44.083230 (+ 753us) client_negotiation.cc:240] Negotiation successful
0925 04:44:44.083807 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5570,"mutex_wait_us":747,"spinlock_wait_cycles":11264,"thread_start_us":83,"threads_started":1}
I20260925 04:44:44.084815 7482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:44.066141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:46181 (local address 127.0.98.65:50061)
0925 04:44:44.066805 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:44.066860 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:44.068451 (+ 1591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:44.069090 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:44.069097 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:44.069630 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:44.070205 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:44.070220 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:44.071556 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:44.071560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:44.073219 (+ 1659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:44.073228 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:44.073964 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:44.073970 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:44.074200 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:44.075237 (+ 1037us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:44.075254 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:44.078189 (+ 2935us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:44:44.081241 (+ 3052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:44.081244 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:44.081249 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:44.081477 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:44.081784 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:44.081787 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:44.081788 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:44.081857 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:44.082319 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:44:44.082322 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:44:44.082477 (+ 155us) client_negotiation.cc:769] Sending connection context
0925 04:44:44.083230 (+ 753us) client_negotiation.cc:240] Negotiation successful
0925 04:44:44.083757 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"mutex_wait_us":308,"spinlock_wait_cycles":35584,"thread_start_us":665,"threads_started":1}
I20260925 04:44:44.085119 7485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:44.066141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:45861 (local address 127.0.98.65:39537)
0925 04:44:44.069067 (+ 2926us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:44.069084 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:44.069187 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:44.069458 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:44.069464 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:44.069762 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:44.070208 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:44.070224 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:44.071605 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:44.071611 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:44.073233 (+ 1622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:44.073246 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:44.074084 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:44.074090 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:44.074279 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:44.075720 (+ 1441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:44.075754 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:44.078502 (+ 2748us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:44:44.080709 (+ 2207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:44.080715 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:44.080720 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:44.081105 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:44.081483 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:44.081488 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:44.081492 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:44.081616 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:44.082171 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:44:44.082175 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:44:44.082482 (+ 307us) client_negotiation.cc:769] Sending connection context
0925 04:44:44.083382 (+ 900us) client_negotiation.cc:240] Negotiation successful
0925 04:44:44.083654 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2824,"mutex_wait_us":276,"spinlock_wait_cycles":10496,"thread_start_us":189,"threads_started":1}
I20260925 04:44:44.086133 7479 heartbeater.cc:347] Connected to a master server at 127.0.98.126:46181
I20260925 04:44:44.086139 7480 heartbeater.cc:347] Connected to a master server at 127.0.98.125:45861
I20260925 04:44:44.086138 7478 heartbeater.cc:347] Connected to a master server at 127.0.98.124:41887
I20260925 04:44:44.086426 7479 heartbeater.cc:464] Registering TS with master...
I20260925 04:44:44.086465 7478 heartbeater.cc:464] Registering TS with master...
I20260925 04:44:44.086426 7480 heartbeater.cc:464] Registering TS with master...
I20260925 04:44:44.087030 7480 heartbeater.cc:511] Master 127.0.98.125:45861 requested a full tablet report, sending...
I20260925 04:44:44.087030 7479 heartbeater.cc:511] Master 127.0.98.126:46181 requested a full tablet report, sending...
I20260925 04:44:44.087646 7478 heartbeater.cc:511] Master 127.0.98.124:41887 requested a full tablet report, sending...
I20260925 04:44:44.088649 7299 ts_manager.cc:194] Registered new tserver with Master: 0bbea834c4b244998ef191efd5bb4173 (127.0.98.65:46321)
I20260925 04:44:44.088706 7116 ts_manager.cc:194] Registered new tserver with Master: 0bbea834c4b244998ef191efd5bb4173 (127.0.98.65:46321)
I20260925 04:44:44.089825 6516 ts_manager.cc:194] Re-registered known tserver with Master: 0bbea834c4b244998ef191efd5bb4173 (127.0.98.65:46321)
I20260925 04:44:44.090480 6516 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.65@KRBTEST.COM'} at 127.0.98.65:50061
I20260925 04:44:44.191625 6975 heartbeater.cc:503] Master 127.0.98.126:46181 was elected leader, sending a full tablet report...
W20260925 04:44:44.195274 7486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:44:44.195570 7486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:44:44.195649 7486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:44:44.199812 7486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:44:44.199895 7486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:44:44.199982 7486 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260925 04:44:44.205457 7486 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:33935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/krb5kdc/oryx_127.0.98.66.keytab
--principal=oryx/127.0.98.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:36667
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=41511
--tserver_master_addrs=127.0.98.126:46181,127.0.98.125:45861,127.0.98.124:41887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7486
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:44:44.206640 7486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:44:44.207654 7486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:44:44.217716 7494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:44.217724 7491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:44.217831 7486 server_base.cc:1034] running on GCE node
W20260925 04:44:44.217730 7492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:44.218456 7486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:44:44.219107 7486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:44:44.220285 7486 hybrid_clock.cc:648] HybridClock initialized: now 1790311484220289 us; error 50 us; skew 500 ppm
Sep 25 04:44:44 dist-test-slave-kghz krb5kdc[6471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311484, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:44:44.223219 7486 init.cc:377] Logged in from keytab as oryx/127.0.98.66@KRBTEST.COM (short username oryx)
I20260925 04:44:44.224087 7486 webserver.cc:460] Webserver started at http://127.0.98.66:41511/ using document root <none> and password file <none>
I20260925 04:44:44.224690 7486 fs_manager.cc:359] Metadata directory not provided
I20260925 04:44:44.224789 7486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:44:44.228904 7486 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 04:44:44.231177 7501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:44.232561 7486 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:44:44.232707 7486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-1/wal
uuid: "49d951568c0a4ae2a35eeedd7daf1646"
format_stamp: "Formatted at 2026-09-25 04:44:42 on dist-test-slave-kghz"
server_key: "53880ea9d4b1d0d760d09a53b696d2ad"
server_key_iv: "142fb452b461d591bf7355e9b4bf7354"
server_key_version: "encryptionkey@0"
I20260925 04:44:44.233321 7486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:44:44.247892 7486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:44:44.251418 7486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:44:44.251643 7486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:44:44.252274 7486 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:44:44.253185 7486 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:44:44.253254 7486 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:44.253325 7486 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:44:44.253365 7486 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:44.262781 7486 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:36667
I20260925 04:44:44.262811 7614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:36667 every 8 connection(s)
I20260925 04:44:44.263929 7486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
I20260925 04:44:44.264281 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 7486
I20260925 04:44:44.264570 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 6846
I20260925 04:44:44.273087 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:38689
--local_ip_for_outbound_sockets=127.0.98.67
--tserver_master_addrs=127.0.98.126:46181
--webserver_port=42029
--webserver_interface=127.0.98.67
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/krb5kdc/oryx_127.0.98.67.keytab
--principal=oryx/127.0.98.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:33935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.126:46181,127.0.98.125:45861
--tserver_master_addrs=127.0.98.126:46181,127.0.98.125:45861,127.0.98.124:41887 with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 04:44:44 dist-test-slave-kghz krb5kdc[6471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311484, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.125@KRBTEST.COM
Sep 25 04:44:44 dist-test-slave-kghz krb5kdc[6471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311484, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.124@KRBTEST.COM
Sep 25 04:44:44 dist-test-slave-kghz krb5kdc[6471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311484, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM
I20260925 04:44:44.284090 7238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:44.266071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:41285 (local address 127.0.98.125:45861)
0925 04:44:44.266207 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:44.266212 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:44.267920 (+ 1708us) server_negotiation.cc:407] Connection header received
0925 04:44:44.269208 (+ 1288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:44.269211 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:44.269266 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:44.269331 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:44.271303 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:44.271992 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:44.273886 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:44.274224 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:44.274331 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:44.279129 (+ 4798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:44.279135 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:44.279141 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:44.279170 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:44.281316 (+ 2146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:44.282068 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:44.282071 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:44.282074 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:44.282165 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:44.282627 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:44.282630 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:44.282632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:44.282888 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:44.282967 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:44.283791 (+ 824us) server_negotiation.cc:299] Negotiation successful
0925 04:44:44.283916 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 04:44:44.284195 6555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:44.267891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:37729 (local address 127.0.98.126:46181)
0925 04:44:44.268029 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:44.268036 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:44.268486 (+ 450us) server_negotiation.cc:407] Connection header received
0925 04:44:44.270634 (+ 2148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:44.270638 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:44.270691 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:44.270745 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:44.272337 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:44.273073 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:44.274692 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:44.275009 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:44.275128 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:44.279267 (+ 4139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:44.279275 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:44.279280 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:44.279315 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:44.281186 (+ 1871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:44.282099 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:44.282106 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:44.282110 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:44.282212 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:44.282644 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:44.282649 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:44.282653 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:44.282908 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:44.283001 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:44.283825 (+ 824us) server_negotiation.cc:299] Negotiation successful
0925 04:44:44.283971 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 04:44:44.285023 7483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:44.266365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:38261 (local address 127.0.98.124:41887)
0925 04:44:44.267648 (+ 1283us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:44.267656 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:44.267918 (+ 262us) server_negotiation.cc:407] Connection header received
0925 04:44:44.269150 (+ 1232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:44.269154 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:44.269217 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:44.269273 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:44.271542 (+ 2269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:44.272397 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:44.273886 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:44.274238 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:44.274331 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:44.279035 (+ 4704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:44.279040 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:44.279044 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:44.279082 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:44.281316 (+ 2234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:44.282068 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:44.282072 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:44.282074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:44.282165 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:44.282634 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:44.282638 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:44.282640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:44.282879 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:44.282967 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:44.284752 (+ 1785us) server_negotiation.cc:299] Negotiation successful
0925 04:44:44.284890 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1162}
I20260925 04:44:44.285511 7621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:44.266611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:45861 (local address 127.0.98.66:41285)
0925 04:44:44.267754 (+ 1143us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:44.267815 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:44.268879 (+ 1064us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:44.269611 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:44.269620 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:44.270439 (+ 819us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:44.271150 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:44.271170 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:44.272133 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:44.272138 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:44.273748 (+ 1610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:44.273759 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:44.274541 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:44.274545 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:44.274850 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:44.275569 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:44.275593 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:44.278938 (+ 3345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:44:44.281447 (+ 2509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:44.281450 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:44.281452 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:44.281947 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:44.282294 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:44.282296 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:44.282298 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:44.282512 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:44.283022 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:44:44.283029 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:44:44.283494 (+ 465us) client_negotiation.cc:769] Sending connection context
0925 04:44:44.284287 (+ 793us) client_negotiation.cc:240] Negotiation successful
0925 04:44:44.284440 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":940,"mutex_wait_us":218,"spinlock_wait_cycles":37888,"thread_start_us":134,"threads_started":1}
I20260925 04:44:44.285558 7619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:44.266611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:41887 (local address 127.0.98.66:38261)
0925 04:44:44.267754 (+ 1143us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:44.267815 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:44.268879 (+ 1064us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:44.269611 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:44.269620 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:44.270453 (+ 833us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:44.271375 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:44.271381 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:44.272518 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:44.272521 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:44.273748 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:44.273759 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:44.274541 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:44.274545 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:44.274845 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:44.275892 (+ 1047us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:44.275906 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:44.278836 (+ 2930us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:44:44.281447 (+ 2611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:44.281450 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:44.281452 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:44.281947 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:44.282292 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:44.282295 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:44.282297 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:44.282512 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:44.283022 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:44:44.283029 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:44:44.283494 (+ 465us) client_negotiation.cc:769] Sending connection context
0925 04:44:44.283718 (+ 224us) client_negotiation.cc:240] Negotiation successful
0925 04:44:44.284059 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1009,"mutex_wait_us":260,"spinlock_wait_cycles":95872,"thread_start_us":128,"threads_started":1}
I20260925 04:44:44.286141 7620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:44.267161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:46181 (local address 127.0.98.66:37729)
0925 04:44:44.268401 (+ 1240us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:44.268415 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:44.270342 (+ 1927us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:44.270875 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:44.270878 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:44.271121 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:44.272169 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:44.272178 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:44.273505 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:44.273512 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:44.274529 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:44.274543 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:44.275278 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:44.275283 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:44.275404 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:44.275870 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:44.275884 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:44.279102 (+ 3218us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:44:44.281340 (+ 2238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:44.281350 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:44.281372 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:44.281968 (+ 596us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:44.282364 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:44.282368 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:44.282372 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:44.282522 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:44.283067 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:44:44.283072 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:44:44.283495 (+ 423us) client_negotiation.cc:769] Sending connection context
0925 04:44:44.284338 (+ 843us) client_negotiation.cc:240] Negotiation successful
0925 04:44:44.284533 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1179,"mutex_wait_us":11,"spinlock_wait_cycles":25216,"thread_start_us":108,"threads_started":1}
I20260925 04:44:44.287067 7615 heartbeater.cc:347] Connected to a master server at 127.0.98.124:41887
I20260925 04:44:44.287348 7615 heartbeater.cc:464] Registering TS with master...
I20260925 04:44:44.287881 7615 heartbeater.cc:511] Master 127.0.98.124:41887 requested a full tablet report, sending...
I20260925 04:44:44.288388 7617 heartbeater.cc:347] Connected to a master server at 127.0.98.125:45861
I20260925 04:44:44.288457 7617 heartbeater.cc:464] Registering TS with master...
I20260925 04:44:44.288631 7617 heartbeater.cc:511] Master 127.0.98.125:45861 requested a full tablet report, sending...
I20260925 04:44:44.288841 7616 heartbeater.cc:347] Connected to a master server at 127.0.98.126:46181
I20260925 04:44:44.288906 7616 heartbeater.cc:464] Registering TS with master...
I20260925 04:44:44.289057 7616 heartbeater.cc:511] Master 127.0.98.126:46181 requested a full tablet report, sending...
I20260925 04:44:44.289165 7299 ts_manager.cc:194] Registered new tserver with Master: 49d951568c0a4ae2a35eeedd7daf1646 (127.0.98.66:36667)
I20260925 04:44:44.289391 7116 ts_manager.cc:194] Registered new tserver with Master: 49d951568c0a4ae2a35eeedd7daf1646 (127.0.98.66:36667)
I20260925 04:44:44.289839 6516 ts_manager.cc:194] Re-registered known tserver with Master: 49d951568c0a4ae2a35eeedd7daf1646 (127.0.98.66:36667)
I20260925 04:44:44.290608 6516 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.66@KRBTEST.COM'} at 127.0.98.66:37729
W20260925 04:44:44.393672 7622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:44:44.394012 7622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:44:44.394119 7622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:44:44.398434 7622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:44:44.398521 7622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:44:44.398610 7622 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260925 04:44:44.404114 7622 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:33935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/krb5kdc/oryx_127.0.98.67.keytab
--principal=oryx/127.0.98.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:38689
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=42029
--tserver_master_addrs=127.0.98.126:46181,127.0.98.125:45861,127.0.98.124:41887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7622
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:44:44.405303 7622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:44:44.406452 7622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:44:44.417023 7627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:44.417126 7628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:44.417299 7622 server_base.cc:1034] running on GCE node
W20260925 04:44:44.417023 7630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:44.417865 7622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:44:44.418607 7622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:44:44.419796 7622 hybrid_clock.cc:648] HybridClock initialized: now 1790311484419780 us; error 32 us; skew 500 ppm
Sep 25 04:44:44 dist-test-slave-kghz krb5kdc[6471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311484, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:44:44.422658 7622 init.cc:377] Logged in from keytab as oryx/127.0.98.67@KRBTEST.COM (short username oryx)
I20260925 04:44:44.423614 7622 webserver.cc:460] Webserver started at http://127.0.98.67:42029/ using document root <none> and password file <none>
I20260925 04:44:44.424142 7622 fs_manager.cc:359] Metadata directory not provided
I20260925 04:44:44.424214 7622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:44:44.428311 7622 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 04:44:44.430624 7637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:44.431988 7622 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 04:44:44.432094 7622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-2/wal
uuid: "1870b79591534da9a311af3678fc937e"
format_stamp: "Formatted at 2026-09-25 04:44:43 on dist-test-slave-kghz"
server_key: "72dab39a63b15b8f02ecc71a8e7025c9"
server_key_iv: "e85a9c7847903b53e8bbc6875921bd1e"
server_key_version: "encryptionkey@0"
I20260925 04:44:44.432672 7622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:44:44.447712 7622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:44:44.451619 7622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:44:44.451841 7622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:44:44.452471 7622 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:44:44.453370 7622 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:44:44.453452 7622 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:44.453545 7622 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:44:44.453593 7622 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:44.463168 7622 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:38689
I20260925 04:44:44.463243 7750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:38689 every 8 connection(s)
I20260925 04:44:44.464331 7622 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
I20260925 04:44:44.471218 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 7622
Sep 25 04:44:44 dist-test-slave-kghz krb5kdc[6471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311484, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.124@KRBTEST.COM
Sep 25 04:44:44 dist-test-slave-kghz krb5kdc[6471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311484, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM
Sep 25 04:44:44 dist-test-slave-kghz krb5kdc[6471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311484, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.125@KRBTEST.COM
I20260925 04:44:44.486711 6555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:44.468052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:43661 (local address 127.0.98.126:46181)
0925 04:44:44.468165 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:44.468169 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:44.468189 (+ 20us) server_negotiation.cc:407] Connection header received
0925 04:44:44.469412 (+ 1223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:44.469416 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:44.469471 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:44.469555 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:44.473508 (+ 3953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:44.474358 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:44.476278 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:44.476616 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:44.476713 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:44.481841 (+ 5128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:44.481848 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:44.481854 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:44.481885 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:44.483857 (+ 1972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:44.484550 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:44.484554 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:44.484556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:44.484646 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:44.485068 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:44.485073 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:44.485076 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:44.485310 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:44.486306 (+ 996us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:44.486383 (+ 77us) server_negotiation.cc:299] Negotiation successful
0925 04:44:44.486518 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 04:44:44.487743 7764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:44.473317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39398 (local address 127.0.98.126:46181)
0925 04:44:44.475418 (+ 2101us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:44.475431 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:44.478063 (+ 2632us) server_negotiation.cc:407] Connection header received
0925 04:44:44.478184 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:44.478187 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:44.478263 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:44.478348 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:44.479472 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:44.480324 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:44.481513 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:44.481973 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:44.482098 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:44.483782 (+ 1684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:44.483786 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:44.483788 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:44.483814 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:44.485534 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:44.486023 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:44.486029 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:44.486034 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:44.486123 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:44.486434 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:44.486438 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:44.486441 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:44.486650 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:44.486762 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:44.487473 (+ 711us) server_negotiation.cc:299] Negotiation successful
0925 04:44:44.487599 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2026,"thread_start_us":72,"threads_started":1}
I20260925 04:44:44.488096 7755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:44.466817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:41887 (local address 127.0.98.67:59989)
0925 04:44:44.467356 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:44.467411 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:44.469200 (+ 1789us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:44.469746 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:44.469751 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:44.470376 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:44.473392 (+ 3016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:44.473402 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:44.474732 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:44.474736 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:44.475884 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:44.475895 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:44.476639 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:44.476643 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:44.476985 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:44.477924 (+ 939us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:44.477941 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:44.480648 (+ 2707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:44:44.485308 (+ 4660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:44.485311 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:44.485314 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:44.485540 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:44.485826 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:44.485829 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:44.485831 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:44.485907 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:44.486456 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:44:44.486459 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:44:44.486557 (+ 98us) client_negotiation.cc:769] Sending connection context
0925 04:44:44.486649 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 04:44:44.486863 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"mutex_wait_us":2438,"spinlock_wait_cycles":22144,"thread_start_us":172,"threads_started":1}
I20260925 04:44:44.488332 7756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:44.466818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:45861 (local address 127.0.98.67:43399)
0925 04:44:44.467654 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:44.467667 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:44.469200 (+ 1533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:44.469746 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:44.469751 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:44.470376 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:44.473217 (+ 2841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:44.473244 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:44.474238 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:44.474241 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:44.475884 (+ 1643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:44.475895 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:44.476639 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:44.476643 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:44.476988 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:44.478079 (+ 1091us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:44.478099 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:44.482548 (+ 4449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:44:44.486190 (+ 3642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:44.486193 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:44.486195 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:44.486449 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:44.486748 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:44.486754 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:44.486756 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:44.486836 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:44.487312 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:44:44.487315 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:44:44.487403 (+ 88us) client_negotiation.cc:769] Sending connection context
0925 04:44:44.487489 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 04:44:44.487608 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":754,"mutex_wait_us":364,"spinlock_wait_cycles":17152,"thread_start_us":116,"threads_started":1}
I20260925 04:44:44.488752 7757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:44.467023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:46181 (local address 127.0.98.67:43661)
0925 04:44:44.468104 (+ 1081us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:44.468118 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:44.469198 (+ 1080us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:44.469744 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:44.469750 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:44.470385 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:44.473217 (+ 2832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:44.473244 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:44.474568 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:44.474571 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:44.476148 (+ 1577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:44.476157 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:44.476844 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:44.476848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:44.476996 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:44.477863 (+ 867us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:44.477883 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:44.481685 (+ 3802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:44:44.483996 (+ 2311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:44.484005 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:44.484029 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:44.484415 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:44.484756 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:44.484761 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:44.484763 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:44.484948 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:44.485429 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:44:44.485435 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:44:44.485807 (+ 372us) client_negotiation.cc:769] Sending connection context
0925 04:44:44.486021 (+ 214us) client_negotiation.cc:240] Negotiation successful
0925 04:44:44.487302 (+ 1281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1017,"mutex_wait_us":2755,"spinlock_wait_cycles":22656,"thread_start_us":84,"threads_started":1}
I20260925 04:44:44.488992 7483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:44.466340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:59989 (local address 127.0.98.124:41887)
0925 04:44:44.466498 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:44.466503 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:44.467521 (+ 1018us) server_negotiation.cc:407] Connection header received
0925 04:44:44.469437 (+ 1916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:44.469440 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:44.469495 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:44.469557 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:44.473546 (+ 3989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:44.474357 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:44.476027 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:44.476374 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:44.476481 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:44.480848 (+ 4367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:44.480852 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:44.480854 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:44.480881 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:44.484221 (+ 3340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:44.485645 (+ 1424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:44.485649 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:44.485650 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:44.485730 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:44.486020 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:44.486024 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:44.486025 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:44.486315 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:44.486411 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:44.488680 (+ 2269us) server_negotiation.cc:299] Negotiation successful
0925 04:44:44.488815 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":62}
I20260925 04:44:44.489001 7238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:44.467934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:43399 (local address 127.0.98.125:45861)
0925 04:44:44.468059 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:44.468064 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:44.468080 (+ 16us) server_negotiation.cc:407] Connection header received
0925 04:44:44.469437 (+ 1357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:44.469440 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:44.469494 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:44.469550 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:44.473428 (+ 3878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:44.474093 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:44.476027 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:44.476389 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:44.476472 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:44.482715 (+ 6243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:44.482722 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:44.482727 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:44.482762 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:44.485181 (+ 2419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:44.486564 (+ 1383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:44.486567 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:44.486569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:44.486648 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:44.486948 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:44.486953 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:44.486957 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:44.487180 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:44.487272 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:44.488672 (+ 1400us) server_negotiation.cc:299] Negotiation successful
0925 04:44:44.488805 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":54}
I20260925 04:44:44.489699 7753 heartbeater.cc:347] Connected to a master server at 127.0.98.125:45861
I20260925 04:44:44.489938 7753 heartbeater.cc:464] Registering TS with master...
I20260925 04:44:44.489955 7752 heartbeater.cc:347] Connected to a master server at 127.0.98.126:46181
I20260925 04:44:44.490402 7752 heartbeater.cc:464] Registering TS with master...
I20260925 04:44:44.489938 7751 heartbeater.cc:347] Connected to a master server at 127.0.98.124:41887
I20260925 04:44:44.490595 7751 heartbeater.cc:464] Registering TS with master...
I20260925 04:44:44.490705 7753 heartbeater.cc:511] Master 127.0.98.125:45861 requested a full tablet report, sending...
I20260925 04:44:44.490728 7751 heartbeater.cc:511] Master 127.0.98.124:41887 requested a full tablet report, sending...
I20260925 04:44:44.491470 7752 heartbeater.cc:511] Master 127.0.98.126:46181 requested a full tablet report, sending...
I20260925 04:44:44.492354 7299 ts_manager.cc:194] Registered new tserver with Master: 1870b79591534da9a311af3678fc937e (127.0.98.67:38689)
I20260925 04:44:44.493096 7116 ts_manager.cc:194] Registered new tserver with Master: 1870b79591534da9a311af3678fc937e (127.0.98.67:38689)
I20260925 04:44:44.493834 7762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:44.473096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49462 (local address 127.0.98.125:45861)
0925 04:44:44.474878 (+ 1782us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:44.474882 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:44.474896 (+ 14us) server_negotiation.cc:407] Connection header received
0925 04:44:44.474932 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:44.474935 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:44.474984 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:44.475069 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:44.476093 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:44.476882 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:44.478333 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:44.478744 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:44.478848 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:44.481585 (+ 2737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:44.481590 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:44.481593 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:44.481622 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:44.491534 (+ 9912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:44.492099 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:44.492103 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:44.492105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:44.492190 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:44.492489 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:44.492492 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:44.492495 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:44.492732 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:44.493449 (+ 717us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:44.493540 (+ 91us) server_negotiation.cc:299] Negotiation successful
0925 04:44:44.493665 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":1705,"thread_start_us":73,"threads_started":1}
I20260925 04:44:44.494076 6515 ts_manager.cc:194] Re-registered known tserver with Master: 1870b79591534da9a311af3678fc937e (127.0.98.67:38689)
I20260925 04:44:44.494792 6515 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.67@KRBTEST.COM'} at 127.0.98.67:43661
I20260925 04:44:44.491793 6516 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39398:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:44:44.496480 6516 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:44:44.496932 7766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:44.474596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45604 (local address 127.0.98.124:41887)
0925 04:44:44.477456 (+ 2860us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:44.477467 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:44.477710 (+ 243us) server_negotiation.cc:407] Connection header received
0925 04:44:44.477748 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:44.477750 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:44.483065 (+ 5315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:44.483166 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:44.484070 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:44.484926 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:44.486118 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:44.490959 (+ 4841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:44.491070 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:44.493023 (+ 1953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:44.493026 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:44.493029 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:44.493064 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:44.494828 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:44.495525 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:44.495528 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:44.495530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:44.495621 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:44.495915 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:44.495920 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:44.495922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:44.496093 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:44.496226 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:44.496468 (+ 242us) server_negotiation.cc:299] Negotiation successful
0925 04:44:44.496698 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":2776,"thread_start_us":77,"threads_started":1}
I20260925 04:44:44.506784 7766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:44.502836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45606 (local address 127.0.98.124:41887)
0925 04:44:44.502954 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:44.502960 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:44.503055 (+ 95us) server_negotiation.cc:407] Connection header received
0925 04:44:44.503210 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:44.503214 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:44.503276 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:44.503368 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:44:44.503627 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:44.504162 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:44.505454 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:44.506088 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:44.506190 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:44.506381 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:44.506567 (+ 186us) server_negotiation.cc:299] Negotiation successful
0925 04:44:44.506646 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 04:44:44.506786 7773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:44.502737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:41887 (local address 127.0.0.1:45606)
0925 04:44:44.502954 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:44.502970 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:44.503102 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:44.503387 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:44.503390 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:44.503406 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:44:44.503525 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:44.503530 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:44.504268 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:44.504275 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:44.505318 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:44.505329 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:44.506328 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:44.506332 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:44.506425 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:44.506457 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 04:44:44.506539 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 04:44:44.506635 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":89,"threads_started":1}
I20260925 04:44:44.507655 7309 raft_consensus.cc:1203] T 00000000000000000000000000000000 P b655e5719e73450cb120a45e38394900 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.8] Dedup: 1.7->[1.8-1.8]
I20260925 04:44:44.510564 7774 sys_catalog.cc:455] T 00000000000000000000000000000000 P b655e5719e73450cb120a45e38394900 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58edb7db5329430685f0d9e7a5098394. Latest consensus state: current_term: 1 leader_uuid: "58edb7db5329430685f0d9e7a5098394" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "58edb7db5329430685f0d9e7a5098394" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46181 } } peers { permanent_uuid: "4f99000937494cc99e7b9bd62901f3f0" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 45861 } attrs { promote: false } } peers { permanent_uuid: "b655e5719e73450cb120a45e38394900" member_type: NON_VOTER last_known_addr { host: "127.0.98.124" port: 41887 } attrs { promote: true } } }
I20260925 04:44:44.510718 7774 sys_catalog.cc:458] T 00000000000000000000000000000000 P b655e5719e73450cb120a45e38394900 [sys.catalog]: This master's current role is: LEARNER
I20260925 04:44:44.512136 7076 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58edb7db5329430685f0d9e7a5098394 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "58edb7db5329430685f0d9e7a5098394" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "58edb7db5329430685f0d9e7a5098394" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46181 } } peers { permanent_uuid: "4f99000937494cc99e7b9bd62901f3f0" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 45861 } attrs { promote: false } } peers { permanent_uuid: "b655e5719e73450cb120a45e38394900" member_type: NON_VOTER last_known_addr { host: "127.0.98.124" port: 41887 } attrs { promote: true } } }
I20260925 04:44:44.512280 7076 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58edb7db5329430685f0d9e7a5098394 [sys.catalog]: This master's current role is: LEADER
I20260925 04:44:44.513895 7769 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 58edb7db5329430685f0d9e7a5098394: attempting to promote NON_VOTER b655e5719e73450cb120a45e38394900 to VOTER
I20260925 04:44:44.514284 7769 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58edb7db5329430685f0d9e7a5098394 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "58edb7db5329430685f0d9e7a5098394" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46181 } } peers { permanent_uuid: "4f99000937494cc99e7b9bd62901f3f0" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 45861 } attrs { promote: false } } peers { permanent_uuid: "b655e5719e73450cb120a45e38394900" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 41887 } attrs { promote: false } }
I20260925 04:44:44.515767 7126 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4f99000937494cc99e7b9bd62901f3f0 [term 1 FOLLOWER]: Refusing update from remote peer 58edb7db5329430685f0d9e7a5098394: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20260925 04:44:44.515970 7773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:44.508721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:36667 (local address 127.0.0.1:41126)
0925 04:44:44.508860 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:44.508874 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:44.508946 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:44.510866 (+ 1920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:44.510869 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:44.510888 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:44:44.510990 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:44.510996 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:44.512702 (+ 1706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:44.512705 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:44.513815 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:44.513824 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:44.515592 (+ 1768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:44.515596 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:44.515676 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:44.515703 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 04:44:44.515754 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 04:44:44.515837 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260925 04:44:44.516248 7777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:44.509300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:38689 (local address 127.0.0.1:43234)
0925 04:44:44.510165 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:44.510182 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:44.510300 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:44.512122 (+ 1822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:44.512125 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:44.512152 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:44:44.512243 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:44.512249 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:44.513697 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:44.513701 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:44.514787 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:44.514800 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:44.515878 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:44.515881 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:44.515963 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:44.515997 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 04:44:44.516063 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 04:44:44.516128 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":796,"thread_start_us":111,"threads_started":1}
I20260925 04:44:44.516307 7780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:44.509738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43234 (local address 127.0.98.67:38689)
0925 04:44:44.511641 (+ 1903us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:44.511654 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:44.511678 (+ 24us) server_negotiation.cc:407] Connection header received
0925 04:44:44.511755 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:44.511761 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:44.511958 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:44.512407 (+ 449us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:44:44.512572 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:44.513570 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:44.514941 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:44.515745 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:44.515877 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:44.516034 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:44.516089 (+ 55us) server_negotiation.cc:299] Negotiation successful
0925 04:44:44.516174 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1779,"thread_start_us":106,"threads_started":1}
I20260925 04:44:44.516278 7076 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 58edb7db5329430685f0d9e7a5098394 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f99000937494cc99e7b9bd62901f3f0" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 45861 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260925 04:44:44.516993 7309 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b655e5719e73450cb120a45e38394900 [term 1 LEARNER]: Refusing update from remote peer 58edb7db5329430685f0d9e7a5098394: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20260925 04:44:44.517161 7776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:44.509296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41126 (local address 127.0.98.66:36667)
0925 04:44:44.509827 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:44.509840 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:44.509864 (+ 24us) server_negotiation.cc:407] Connection header received
0925 04:44:44.510439 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:44.510445 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:44.510686 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:44.510889 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:44:44.511098 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:44.512252 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:44.514632 (+ 2380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:44.515449 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:44.516569 (+ 1120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:44.516742 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:44.516798 (+ 56us) server_negotiation.cc:299] Negotiation successful
0925 04:44:44.516892 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":93,"threads_started":1}
I20260925 04:44:44.518260 7076 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 58edb7db5329430685f0d9e7a5098394 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b655e5719e73450cb120a45e38394900" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 41887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260925 04:44:44.518747 7076 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 58edb7db5329430685f0d9e7a5098394 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 7 to 10, b655e5719e73450cb120a45e38394900 (127.0.98.124) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "58edb7db5329430685f0d9e7a5098394" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46181 } } peers { permanent_uuid: "4f99000937494cc99e7b9bd62901f3f0" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 45861 } attrs { promote: false } } peers { permanent_uuid: "b655e5719e73450cb120a45e38394900" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 41887 } attrs { promote: false } } }
I20260925 04:44:44.520066 7076 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58edb7db5329430685f0d9e7a5098394 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "58edb7db5329430685f0d9e7a5098394" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "58edb7db5329430685f0d9e7a5098394" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46181 } } peers { permanent_uuid: "4f99000937494cc99e7b9bd62901f3f0" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 45861 } attrs { promote: false } } peers { permanent_uuid: "b655e5719e73450cb120a45e38394900" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 41887 } attrs { promote: false } } }
I20260925 04:44:44.520190 7076 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58edb7db5329430685f0d9e7a5098394 [sys.catalog]: This master's current role is: LEADER
I20260925 04:44:44.519759 7685 tablet_service.cc:1467] Processing CreateTablet for tablet 6f139ed25ebc4639be4a43ce348255a7 (DEFAULT_TABLE table=test-table [id=e386a08edc2840948ca99154b8679699]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:44:44.520004 7549 tablet_service.cc:1467] Processing CreateTablet for tablet 6f139ed25ebc4639be4a43ce348255a7 (DEFAULT_TABLE table=test-table [id=e386a08edc2840948ca99154b8679699]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:44:44.520309 7076 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58edb7db5329430685f0d9e7a5098394 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "58edb7db5329430685f0d9e7a5098394" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "58edb7db5329430685f0d9e7a5098394" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46181 } } peers { permanent_uuid: "4f99000937494cc99e7b9bd62901f3f0" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 45861 } attrs { promote: false } } peers { permanent_uuid: "b655e5719e73450cb120a45e38394900" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 41887 } attrs { promote: false } } }
I20260925 04:44:44.520393 7076 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58edb7db5329430685f0d9e7a5098394 [sys.catalog]: This master's current role is: LEADER
I20260925 04:44:44.520991 7685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f139ed25ebc4639be4a43ce348255a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:44:44.520998 7549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f139ed25ebc4639be4a43ce348255a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:44:44.520963 7126 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4f99000937494cc99e7b9bd62901f3f0 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 7 to 10, b655e5719e73450cb120a45e38394900 (127.0.98.124) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "58edb7db5329430685f0d9e7a5098394" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46181 } } peers { permanent_uuid: "4f99000937494cc99e7b9bd62901f3f0" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 45861 } attrs { promote: false } } peers { permanent_uuid: "b655e5719e73450cb120a45e38394900" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 41887 } attrs { promote: false } } }
I20260925 04:44:44.522698 7769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58edb7db5329430685f0d9e7a5098394 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "58edb7db5329430685f0d9e7a5098394" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "58edb7db5329430685f0d9e7a5098394" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46181 } } peers { permanent_uuid: "4f99000937494cc99e7b9bd62901f3f0" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 45861 } attrs { promote: false } } peers { permanent_uuid: "b655e5719e73450cb120a45e38394900" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 41887 } attrs { promote: false } } }
I20260925 04:44:44.522812 7769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58edb7db5329430685f0d9e7a5098394 [sys.catalog]: This master's current role is: LEADER
I20260925 04:44:44.523286 7309 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b655e5719e73450cb120a45e38394900 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 7 to 10, b655e5719e73450cb120a45e38394900 (127.0.98.124) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "58edb7db5329430685f0d9e7a5098394" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46181 } } peers { permanent_uuid: "4f99000937494cc99e7b9bd62901f3f0" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 45861 } attrs { promote: false } } peers { permanent_uuid: "b655e5719e73450cb120a45e38394900" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 41887 } attrs { promote: false } } }
I20260925 04:44:44.524595 7774 sys_catalog.cc:455] T 00000000000000000000000000000000 P b655e5719e73450cb120a45e38394900 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "58edb7db5329430685f0d9e7a5098394" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "58edb7db5329430685f0d9e7a5098394" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46181 } } peers { permanent_uuid: "4f99000937494cc99e7b9bd62901f3f0" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 45861 } attrs { promote: false } } peers { permanent_uuid: "b655e5719e73450cb120a45e38394900" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 41887 } attrs { promote: false } } }
I20260925 04:44:44.524706 7774 sys_catalog.cc:458] T 00000000000000000000000000000000 P b655e5719e73450cb120a45e38394900 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:44:44.528141 7779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:44.509709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:46321 (local address 127.0.0.1:46872)
0925 04:44:44.512068 (+ 2359us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:44.512082 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:44.512170 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:44.512988 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:44.512992 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:44.513007 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:44:44.513081 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:44.513086 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:44.525633 (+ 12547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:44.525639 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:44.526676 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:44.526687 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:44.527719 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:44.527723 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:44.527812 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:44.527835 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 04:44:44.527902 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 04:44:44.527987 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2301,"spinlock_wait_cycles":384,"thread_start_us":87,"threads_started":1}
I20260925 04:44:44.528162 7782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:44.510096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46872 (local address 127.0.98.65:46321)
0925 04:44:44.512415 (+ 2319us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:44.512426 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:44.512449 (+ 23us) server_negotiation.cc:407] Connection header received
0925 04:44:44.512519 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:44.512523 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:44.512719 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:44.512853 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:44:44.513746 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:44.525478 (+ 11732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:44.526810 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:44.527551 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:44.527720 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:44.527886 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:44.527938 (+ 52us) server_negotiation.cc:299] Negotiation successful
0925 04:44:44.528026 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2221,"thread_start_us":109,"threads_started":1}
I20260925 04:44:44.529699 7787 tablet_bootstrap.cc:492] T 6f139ed25ebc4639be4a43ce348255a7 P 1870b79591534da9a311af3678fc937e: Bootstrap starting.
I20260925 04:44:44.530618 7785 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f99000937494cc99e7b9bd62901f3f0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "58edb7db5329430685f0d9e7a5098394" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "58edb7db5329430685f0d9e7a5098394" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46181 } } peers { permanent_uuid: "4f99000937494cc99e7b9bd62901f3f0" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 45861 } attrs { promote: false } } peers { permanent_uuid: "b655e5719e73450cb120a45e38394900" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 41887 } attrs { promote: false } } }
I20260925 04:44:44.530767 7785 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f99000937494cc99e7b9bd62901f3f0 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:44:44.531654 7412 tablet_service.cc:1467] Processing CreateTablet for tablet 6f139ed25ebc4639be4a43ce348255a7 (DEFAULT_TABLE table=test-table [id=e386a08edc2840948ca99154b8679699]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:44:44.532769 7412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f139ed25ebc4639be4a43ce348255a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:44:44.533087 7787 tablet_bootstrap.cc:654] T 6f139ed25ebc4639be4a43ce348255a7 P 1870b79591534da9a311af3678fc937e: Neither blocks nor log segments found. Creating new log.
I20260925 04:44:44.533965 7787 log.cc:824] T 6f139ed25ebc4639be4a43ce348255a7 P 1870b79591534da9a311af3678fc937e: Log is configured to *not* fsync() on all Append() calls
I20260925 04:44:44.534776 7786 tablet_bootstrap.cc:492] T 6f139ed25ebc4639be4a43ce348255a7 P 49d951568c0a4ae2a35eeedd7daf1646: Bootstrap starting.
I20260925 04:44:44.536336 7787 tablet_bootstrap.cc:492] T 6f139ed25ebc4639be4a43ce348255a7 P 1870b79591534da9a311af3678fc937e: No bootstrap required, opened a new log
I20260925 04:44:44.536648 7787 ts_tablet_manager.cc:1397] T 6f139ed25ebc4639be4a43ce348255a7 P 1870b79591534da9a311af3678fc937e: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260925 04:44:44.537685 7786 tablet_bootstrap.cc:654] T 6f139ed25ebc4639be4a43ce348255a7 P 49d951568c0a4ae2a35eeedd7daf1646: Neither blocks nor log segments found. Creating new log.
I20260925 04:44:44.538620 7786 log.cc:824] T 6f139ed25ebc4639be4a43ce348255a7 P 49d951568c0a4ae2a35eeedd7daf1646: Log is configured to *not* fsync() on all Append() calls
I20260925 04:44:44.540597 7787 raft_consensus.cc:348] T 6f139ed25ebc4639be4a43ce348255a7 P 1870b79591534da9a311af3678fc937e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49d951568c0a4ae2a35eeedd7daf1646" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 36667 } } peers { permanent_uuid: "0bbea834c4b244998ef191efd5bb4173" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46321 } } peers { permanent_uuid: "1870b79591534da9a311af3678fc937e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38689 } }
I20260925 04:44:44.540848 7787 raft_consensus.cc:374] T 6f139ed25ebc4639be4a43ce348255a7 P 1870b79591534da9a311af3678fc937e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:44:44.540900 7787 raft_consensus.cc:729] T 6f139ed25ebc4639be4a43ce348255a7 P 1870b79591534da9a311af3678fc937e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1870b79591534da9a311af3678fc937e, State: Initialized, Role: FOLLOWER
I20260925 04:44:44.540905 7786 tablet_bootstrap.cc:492] T 6f139ed25ebc4639be4a43ce348255a7 P 49d951568c0a4ae2a35eeedd7daf1646: No bootstrap required, opened a new log
I20260925 04:44:44.541090 7786 ts_tablet_manager.cc:1397] T 6f139ed25ebc4639be4a43ce348255a7 P 49d951568c0a4ae2a35eeedd7daf1646: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 04:44:44.541404 7787 consensus_queue.cc:260] T 6f139ed25ebc4639be4a43ce348255a7 P 1870b79591534da9a311af3678fc937e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49d951568c0a4ae2a35eeedd7daf1646" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 36667 } } peers { permanent_uuid: "0bbea834c4b244998ef191efd5bb4173" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46321 } } peers { permanent_uuid: "1870b79591534da9a311af3678fc937e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38689 } }
I20260925 04:44:44.542016 7790 tablet_bootstrap.cc:492] T 6f139ed25ebc4639be4a43ce348255a7 P 0bbea834c4b244998ef191efd5bb4173: Bootstrap starting.
I20260925 04:44:44.542449 7787 ts_tablet_manager.cc:1428] T 6f139ed25ebc4639be4a43ce348255a7 P 1870b79591534da9a311af3678fc937e: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260925 04:44:44.544564 7790 tablet_bootstrap.cc:654] T 6f139ed25ebc4639be4a43ce348255a7 P 0bbea834c4b244998ef191efd5bb4173: Neither blocks nor log segments found. Creating new log.
I20260925 04:44:44.545034 7752 heartbeater.cc:503] Master 127.0.98.126:46181 was elected leader, sending a full tablet report...
I20260925 04:44:44.545430 7790 log.cc:824] T 6f139ed25ebc4639be4a43ce348255a7 P 0bbea834c4b244998ef191efd5bb4173: Log is configured to *not* fsync() on all Append() calls
I20260925 04:44:44.545454 7786 raft_consensus.cc:348] T 6f139ed25ebc4639be4a43ce348255a7 P 49d951568c0a4ae2a35eeedd7daf1646 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49d951568c0a4ae2a35eeedd7daf1646" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 36667 } } peers { permanent_uuid: "0bbea834c4b244998ef191efd5bb4173" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46321 } } peers { permanent_uuid: "1870b79591534da9a311af3678fc937e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38689 } }
I20260925 04:44:44.545738 7786 raft_consensus.cc:374] T 6f139ed25ebc4639be4a43ce348255a7 P 49d951568c0a4ae2a35eeedd7daf1646 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:44:44.545799 7786 raft_consensus.cc:729] T 6f139ed25ebc4639be4a43ce348255a7 P 49d951568c0a4ae2a35eeedd7daf1646 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49d951568c0a4ae2a35eeedd7daf1646, State: Initialized, Role: FOLLOWER
I20260925 04:44:44.546300 7786 consensus_queue.cc:260] T 6f139ed25ebc4639be4a43ce348255a7 P 49d951568c0a4ae2a35eeedd7daf1646 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49d951568c0a4ae2a35eeedd7daf1646" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 36667 } } peers { permanent_uuid: "0bbea834c4b244998ef191efd5bb4173" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46321 } } peers { permanent_uuid: "1870b79591534da9a311af3678fc937e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38689 } }
I20260925 04:44:44.547333 7786 ts_tablet_manager.cc:1428] T 6f139ed25ebc4639be4a43ce348255a7 P 49d951568c0a4ae2a35eeedd7daf1646: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 04:44:44.548617 7616 heartbeater.cc:503] Master 127.0.98.126:46181 was elected leader, sending a full tablet report...
I20260925 04:44:44.549065 7790 tablet_bootstrap.cc:492] T 6f139ed25ebc4639be4a43ce348255a7 P 0bbea834c4b244998ef191efd5bb4173: No bootstrap required, opened a new log
I20260925 04:44:44.549280 7790 ts_tablet_manager.cc:1397] T 6f139ed25ebc4639be4a43ce348255a7 P 0bbea834c4b244998ef191efd5bb4173: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260925 04:44:44.553813 7790 raft_consensus.cc:348] T 6f139ed25ebc4639be4a43ce348255a7 P 0bbea834c4b244998ef191efd5bb4173 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49d951568c0a4ae2a35eeedd7daf1646" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 36667 } } peers { permanent_uuid: "0bbea834c4b244998ef191efd5bb4173" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46321 } } peers { permanent_uuid: "1870b79591534da9a311af3678fc937e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38689 } }
I20260925 04:44:44.554072 7790 raft_consensus.cc:374] T 6f139ed25ebc4639be4a43ce348255a7 P 0bbea834c4b244998ef191efd5bb4173 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:44:44.554142 7790 raft_consensus.cc:729] T 6f139ed25ebc4639be4a43ce348255a7 P 0bbea834c4b244998ef191efd5bb4173 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bbea834c4b244998ef191efd5bb4173, State: Initialized, Role: FOLLOWER
I20260925 04:44:44.554647 7790 consensus_queue.cc:260] T 6f139ed25ebc4639be4a43ce348255a7 P 0bbea834c4b244998ef191efd5bb4173 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49d951568c0a4ae2a35eeedd7daf1646" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 36667 } } peers { permanent_uuid: "0bbea834c4b244998ef191efd5bb4173" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46321 } } peers { permanent_uuid: "1870b79591534da9a311af3678fc937e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38689 } }
I20260925 04:44:44.555323 7479 heartbeater.cc:503] Master 127.0.98.126:46181 was elected leader, sending a full tablet report...
I20260925 04:44:44.555652 7790 ts_tablet_manager.cc:1428] T 6f139ed25ebc4639be4a43ce348255a7 P 0bbea834c4b244998ef191efd5bb4173: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260925 04:44:44.566907 7481 tablet.cc:2369] T 6f139ed25ebc4639be4a43ce348255a7 P 0bbea834c4b244998ef191efd5bb4173: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:44:44.593878 7793 raft_consensus.cc:482] T 6f139ed25ebc4639be4a43ce348255a7 P 49d951568c0a4ae2a35eeedd7daf1646 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:44:44.594115 7793 raft_consensus.cc:504] T 6f139ed25ebc4639be4a43ce348255a7 P 49d951568c0a4ae2a35eeedd7daf1646 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49d951568c0a4ae2a35eeedd7daf1646" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 36667 } } peers { permanent_uuid: "0bbea834c4b244998ef191efd5bb4173" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46321 } } peers { permanent_uuid: "1870b79591534da9a311af3678fc937e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38689 } }
I20260925 04:44:44.595397 7793 leader_election.cc:288] T 6f139ed25ebc4639be4a43ce348255a7 P 49d951568c0a4ae2a35eeedd7daf1646 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0bbea834c4b244998ef191efd5bb4173 (127.0.98.65:46321), 1870b79591534da9a311af3678fc937e (127.0.98.67:38689)
I20260925 04:44:44.600987 7780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:44.595775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:43393 (local address 127.0.98.67:38689)
0925 04:44:44.595891 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:44.595895 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:44.595912 (+ 17us) server_negotiation.cc:407] Connection header received
0925 04:44:44.596009 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:44.596012 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:44.596057 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:44.596143 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:44:44.596560 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:44.597357 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:44.599582 (+ 2225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:44.600193 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:44.600313 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:44.600380 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:44.600765 (+ 385us) server_negotiation.cc:299] Negotiation successful
0925 04:44:44.600851 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 04:44:44.601009 7619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:44.595574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:46321 (local address 127.0.98.66:37001)
0925 04:44:44.595789 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:44.595806 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:44.595900 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:44.596169 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:44.596171 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:44.596193 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:44:44.596423 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:44.596430 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:44.597504 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:44.597508 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:44.599418 (+ 1910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:44.599430 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:44.600359 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:44.600364 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:44.600513 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:44.600621 (+ 108us) client_negotiation.cc:769] Sending connection context
0925 04:44:44.600707 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 04:44:44.600830 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":220,"spinlock_wait_cycles":8192}
I20260925 04:44:44.601023 7620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:44.595671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:38689 (local address 127.0.98.66:43393)
0925 04:44:44.595789 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:44.595806 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:44.595900 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:44.596168 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:44.596171 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:44.596196 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:44:44.596426 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:44.596431 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:44.597504 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:44.597508 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:44.599418 (+ 1910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:44.599430 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:44.600359 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:44.600364 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:44.600504 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:44.600623 (+ 119us) client_negotiation.cc:769] Sending connection context
0925 04:44:44.600707 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 04:44:44.600830 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71,"mutex_wait_us":219,"spinlock_wait_cycles":9088}
I20260925 04:44:44.601024 7782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:44.595695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:37001 (local address 127.0.98.65:46321)
0925 04:44:44.595921 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:44.595926 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:44.595942 (+ 16us) server_negotiation.cc:407] Connection header received
0925 04:44:44.596000 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:44.596003 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:44.596067 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:44.596143 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:44:44.596547 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:44.597372 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:44.599582 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:44.600228 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:44.600323 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:44.600381 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:44.600766 (+ 385us) server_negotiation.cc:299] Negotiation successful
0925 04:44:44.600851 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147}
I20260925 04:44:44.602087 7705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f139ed25ebc4639be4a43ce348255a7" candidate_uuid: "49d951568c0a4ae2a35eeedd7daf1646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1870b79591534da9a311af3678fc937e" is_pre_election: true
I20260925 04:44:44.602092 7432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f139ed25ebc4639be4a43ce348255a7" candidate_uuid: "49d951568c0a4ae2a35eeedd7daf1646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bbea834c4b244998ef191efd5bb4173" is_pre_election: true
I20260925 04:44:44.602418 7432 raft_consensus.cc:2454] T 6f139ed25ebc4639be4a43ce348255a7 P 0bbea834c4b244998ef191efd5bb4173 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49d951568c0a4ae2a35eeedd7daf1646 in term 0.
I20260925 04:44:44.602421 7705 raft_consensus.cc:2454] T 6f139ed25ebc4639be4a43ce348255a7 P 1870b79591534da9a311af3678fc937e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49d951568c0a4ae2a35eeedd7daf1646 in term 0.
I20260925 04:44:44.602921 7502 leader_election.cc:302] T 6f139ed25ebc4639be4a43ce348255a7 P 49d951568c0a4ae2a35eeedd7daf1646 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1870b79591534da9a311af3678fc937e, 49d951568c0a4ae2a35eeedd7daf1646; no voters:
I20260925 04:44:44.603224 7793 raft_consensus.cc:2781] T 6f139ed25ebc4639be4a43ce348255a7 P 49d951568c0a4ae2a35eeedd7daf1646 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:44:44.603322 7793 raft_consensus.cc:482] T 6f139ed25ebc4639be4a43ce348255a7 P 49d951568c0a4ae2a35eeedd7daf1646 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:44:44.603379 7793 raft_consensus.cc:3037] T 6f139ed25ebc4639be4a43ce348255a7 P 49d951568c0a4ae2a35eeedd7daf1646 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:44:44.604538 7793 raft_consensus.cc:504] T 6f139ed25ebc4639be4a43ce348255a7 P 49d951568c0a4ae2a35eeedd7daf1646 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49d951568c0a4ae2a35eeedd7daf1646" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 36667 } } peers { permanent_uuid: "0bbea834c4b244998ef191efd5bb4173" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46321 } } peers { permanent_uuid: "1870b79591534da9a311af3678fc937e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38689 } }
I20260925 04:44:44.604908 7793 leader_election.cc:288] T 6f139ed25ebc4639be4a43ce348255a7 P 49d951568c0a4ae2a35eeedd7daf1646 [CANDIDATE]: Term 1 election: Requested vote from peers 0bbea834c4b244998ef191efd5bb4173 (127.0.98.65:46321), 1870b79591534da9a311af3678fc937e (127.0.98.67:38689)
I20260925 04:44:44.605311 7432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f139ed25ebc4639be4a43ce348255a7" candidate_uuid: "49d951568c0a4ae2a35eeedd7daf1646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bbea834c4b244998ef191efd5bb4173"
I20260925 04:44:44.605455 7432 raft_consensus.cc:3037] T 6f139ed25ebc4639be4a43ce348255a7 P 0bbea834c4b244998ef191efd5bb4173 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:44:44.605548 7705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f139ed25ebc4639be4a43ce348255a7" candidate_uuid: "49d951568c0a4ae2a35eeedd7daf1646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1870b79591534da9a311af3678fc937e"
I20260925 04:44:44.605734 7705 raft_consensus.cc:3037] T 6f139ed25ebc4639be4a43ce348255a7 P 1870b79591534da9a311af3678fc937e [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:44:44.606669 7432 raft_consensus.cc:2454] T 6f139ed25ebc4639be4a43ce348255a7 P 0bbea834c4b244998ef191efd5bb4173 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49d951568c0a4ae2a35eeedd7daf1646 in term 1.
I20260925 04:44:44.606959 7705 raft_consensus.cc:2454] T 6f139ed25ebc4639be4a43ce348255a7 P 1870b79591534da9a311af3678fc937e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49d951568c0a4ae2a35eeedd7daf1646 in term 1.
I20260925 04:44:44.606980 7504 leader_election.cc:302] T 6f139ed25ebc4639be4a43ce348255a7 P 49d951568c0a4ae2a35eeedd7daf1646 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bbea834c4b244998ef191efd5bb4173, 49d951568c0a4ae2a35eeedd7daf1646; no voters:
I20260925 04:44:44.607162 7793 raft_consensus.cc:2781] T 6f139ed25ebc4639be4a43ce348255a7 P 49d951568c0a4ae2a35eeedd7daf1646 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:44:44.607405 7793 raft_consensus.cc:686] T 6f139ed25ebc4639be4a43ce348255a7 P 49d951568c0a4ae2a35eeedd7daf1646 [term 1 LEADER]: Becoming Leader. State: Replica: 49d951568c0a4ae2a35eeedd7daf1646, State: Running, Role: LEADER
I20260925 04:44:44.607745 7793 consensus_queue.cc:237] T 6f139ed25ebc4639be4a43ce348255a7 P 49d951568c0a4ae2a35eeedd7daf1646 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49d951568c0a4ae2a35eeedd7daf1646" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 36667 } } peers { permanent_uuid: "0bbea834c4b244998ef191efd5bb4173" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46321 } } peers { permanent_uuid: "1870b79591534da9a311af3678fc937e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38689 } }
I20260925 04:44:44.611529 6516 catalog_manager.cc:5495] T 6f139ed25ebc4639be4a43ce348255a7 P 49d951568c0a4ae2a35eeedd7daf1646 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49d951568c0a4ae2a35eeedd7daf1646 (127.0.98.66). New cstate: current_term: 1 leader_uuid: "49d951568c0a4ae2a35eeedd7daf1646" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49d951568c0a4ae2a35eeedd7daf1646" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 36667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0bbea834c4b244998ef191efd5bb4173" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1870b79591534da9a311af3678fc937e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38689 } health_report { overall_health: UNKNOWN } } }
I20260925 04:44:44.645931 7776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:44.641787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41128 (local address 127.0.98.66:36667)
0925 04:44:44.641936 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:44.641940 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:44.641958 (+ 18us) server_negotiation.cc:407] Connection header received
0925 04:44:44.642156 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:44.642160 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:44.642386 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:44.642545 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:44:44.642840 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:44.643368 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:44.644272 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:44.644512 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:44.644629 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:44.644979 (+ 350us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:44:44.645471 (+ 492us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:44:44.645611 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:44.645684 (+ 73us) server_negotiation.cc:299] Negotiation successful
0925 04:44:44.645748 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 04:44:44.652776 7432 raft_consensus.cc:1261] T 6f139ed25ebc4639be4a43ce348255a7 P 0bbea834c4b244998ef191efd5bb4173 [term 1 FOLLOWER]: Refusing update from remote peer 49d951568c0a4ae2a35eeedd7daf1646: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:44:44.652774 7705 raft_consensus.cc:1261] T 6f139ed25ebc4639be4a43ce348255a7 P 1870b79591534da9a311af3678fc937e [term 1 FOLLOWER]: Refusing update from remote peer 49d951568c0a4ae2a35eeedd7daf1646: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:44:44.653765 7795 consensus_queue.cc:1035] T 6f139ed25ebc4639be4a43ce348255a7 P 49d951568c0a4ae2a35eeedd7daf1646 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bbea834c4b244998ef191efd5bb4173" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:44:44.653995 7793 consensus_queue.cc:1035] T 6f139ed25ebc4639be4a43ce348255a7 P 49d951568c0a4ae2a35eeedd7daf1646 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1870b79591534da9a311af3678fc937e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:44:44.661077 7798 mvcc.cc:205] Tried to move back new op lower bound from 7333115841128775680 to 7333115840955949056. Current Snapshot: MvccSnapshot[applied={T|T < 7333115841128775680}]
I20260925 04:44:44.663125 7799 mvcc.cc:205] Tried to move back new op lower bound from 7333115841128775680 to 7333115840955949056. Current Snapshot: MvccSnapshot[applied={T|T < 7333115841128775680}]
I20260925 04:44:44.676072 6516 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39398:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:44:44.676273 6516 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39398:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:44:44.681058 6516 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 58edb7db5329430685f0d9e7a5098394: Sending DeleteTablet for 3 replicas of tablet 6f139ed25ebc4639be4a43ce348255a7
I20260925 04:44:44.681913 7549 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f139ed25ebc4639be4a43ce348255a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:44:44 UTC) from {username='oryx', principal='oryx/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:41126
I20260925 04:44:44.682147 7412 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f139ed25ebc4639be4a43ce348255a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:44:44 UTC) from {username='oryx', principal='oryx/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:46872
I20260925 04:44:44.682588 7685 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f139ed25ebc4639be4a43ce348255a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:44:44 UTC) from {username='oryx', principal='oryx/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:43234
I20260925 04:44:44.683060 7810 tablet_replica.cc:330] T 6f139ed25ebc4639be4a43ce348255a7 P 1870b79591534da9a311af3678fc937e: stopping tablet replica
I20260925 04:44:44.683465 7810 raft_consensus.cc:2229] T 6f139ed25ebc4639be4a43ce348255a7 P 1870b79591534da9a311af3678fc937e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:44:44.683818 7810 raft_consensus.cc:2258] T 6f139ed25ebc4639be4a43ce348255a7 P 1870b79591534da9a311af3678fc937e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:44:44.685173 7810 ts_tablet_manager.cc:1905] T 6f139ed25ebc4639be4a43ce348255a7 P 1870b79591534da9a311af3678fc937e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:44:44.685246 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 6484
I20260925 04:44:44.688936 7810 ts_tablet_manager.cc:1918] T 6f139ed25ebc4639be4a43ce348255a7 P 1870b79591534da9a311af3678fc937e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:44:44.689061 7810 log.cc:1196] T 6f139ed25ebc4639be4a43ce348255a7 P 1870b79591534da9a311af3678fc937e: Deleting WAL directory at /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-2/wal/wals/6f139ed25ebc4639be4a43ce348255a7
I20260925 04:44:44.689388 7810 ts_tablet_manager.cc:1939] T 6f139ed25ebc4639be4a43ce348255a7 P 1870b79591534da9a311af3678fc937e: Deleting consensus metadata
I20260925 04:44:44.691164 7809 tablet_replica.cc:330] T 6f139ed25ebc4639be4a43ce348255a7 P 0bbea834c4b244998ef191efd5bb4173: stopping tablet replica
I20260925 04:44:44.691545 7809 raft_consensus.cc:2229] T 6f139ed25ebc4639be4a43ce348255a7 P 0bbea834c4b244998ef191efd5bb4173 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:44:44.691946 7809 raft_consensus.cc:2258] T 6f139ed25ebc4639be4a43ce348255a7 P 0bbea834c4b244998ef191efd5bb4173 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:44:44.692339 7808 tablet_replica.cc:330] T 6f139ed25ebc4639be4a43ce348255a7 P 49d951568c0a4ae2a35eeedd7daf1646: stopping tablet replica
I20260925 04:44:44.692644 7808 raft_consensus.cc:2229] T 6f139ed25ebc4639be4a43ce348255a7 P 49d951568c0a4ae2a35eeedd7daf1646 [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:44:44.694181 7809 ts_tablet_manager.cc:1905] T 6f139ed25ebc4639be4a43ce348255a7 P 0bbea834c4b244998ef191efd5bb4173: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 04:44:44.695247 7641 connection.cc:535] server connection from 127.0.0.1:43234 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 04:44:44.696992 7809 ts_tablet_manager.cc:1918] T 6f139ed25ebc4639be4a43ce348255a7 P 0bbea834c4b244998ef191efd5bb4173: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:44:44.697094 7809 log.cc:1196] T 6f139ed25ebc4639be4a43ce348255a7 P 0bbea834c4b244998ef191efd5bb4173: Deleting WAL directory at /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-0/wal/wals/6f139ed25ebc4639be4a43ce348255a7
I20260925 04:44:44.697402 7809 ts_tablet_manager.cc:1939] T 6f139ed25ebc4639be4a43ce348255a7 P 0bbea834c4b244998ef191efd5bb4173: Deleting consensus metadata
I20260925 04:44:44.698426 7808 raft_consensus.cc:2258] T 6f139ed25ebc4639be4a43ce348255a7 P 49d951568c0a4ae2a35eeedd7daf1646 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260925 04:44:44.699373 7368 connection.cc:441] server connection from 127.0.0.1:46872 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46872 (request call id 1) could send its response
I20260925 04:44:44.701758 7762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:44.683497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49478 (local address 127.0.98.125:45861)
0925 04:44:44.683628 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:44.683633 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:44.690824 (+ 7191us) server_negotiation.cc:407] Connection header received
0925 04:44:44.691011 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:44.691017 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:44.691091 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:44.691178 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:44.692500 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:44.693463 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:44.694895 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:44.695257 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:44.695343 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:44.697831 (+ 2488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:44.697835 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:44.697838 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:44.697871 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:44.699756 (+ 1885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:44.700288 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:44.700292 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:44.700294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:44.700386 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:44.700703 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:44.700708 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:44.700710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:44.700967 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:44.701090 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:44.701453 (+ 363us) server_negotiation.cc:299] Negotiation successful
0925 04:44:44.701600 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 04:44:44.701807 7808 ts_tablet_manager.cc:1905] T 6f139ed25ebc4639be4a43ce348255a7 P 49d951568c0a4ae2a35eeedd7daf1646: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:44:44.705741 7808 ts_tablet_manager.cc:1918] T 6f139ed25ebc4639be4a43ce348255a7 P 49d951568c0a4ae2a35eeedd7daf1646: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:44:44.705862 7808 log.cc:1196] T 6f139ed25ebc4639be4a43ce348255a7 P 49d951568c0a4ae2a35eeedd7daf1646: Deleting WAL directory at /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-1/wal/wals/6f139ed25ebc4639be4a43ce348255a7
I20260925 04:44:44.706193 7808 ts_tablet_manager.cc:1939] T 6f139ed25ebc4639be4a43ce348255a7 P 49d951568c0a4ae2a35eeedd7daf1646: Deleting consensus metadata
W20260925 04:44:44.707531 7503 connection.cc:441] server connection from 127.0.0.1:41126 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41126 (request call id 1) could send its response
I20260925 04:44:44.712687 7766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:44.683892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45616 (local address 127.0.98.124:41887)
0925 04:44:44.683998 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:44.684002 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:44.702713 (+ 18711us) server_negotiation.cc:407] Connection header received
0925 04:44:44.702865 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:44.702868 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:44.702935 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:44.702988 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:44.703847 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:44.704691 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:44.705913 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:44.706281 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:44.706370 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:44.708347 (+ 1977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:44.708352 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:44.708355 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:44.708387 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:44.710211 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:44.711261 (+ 1050us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:44.711265 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:44.711267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:44.711354 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:44.711667 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:44.711670 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:44.711672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:44.711889 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:44.711995 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:44.712346 (+ 351us) server_negotiation.cc:299] Negotiation successful
0925 04:44:44.712503 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":88,"server-negotiator.queue_time_us":41,"spinlock_wait_cycles":14464}
I20260925 04:44:44.715998 7238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:44.701029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49482 (local address 127.0.98.125:45861)
0925 04:44:44.701162 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:44.701166 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:44.702675 (+ 1509us) server_negotiation.cc:407] Connection header received
0925 04:44:44.704555 (+ 1880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:44.704558 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:44.704600 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:44.704644 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:44.705566 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:44.706323 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:44.709456 (+ 3133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:44.709824 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:44.709905 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:44.711703 (+ 1798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:44.711707 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:44.711710 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:44.711737 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:44.714002 (+ 2265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:44.714616 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:44.714623 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:44.714625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:44.714715 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:44.715021 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:44.715025 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:44.715026 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:44.715242 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:44.715354 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:44.715720 (+ 366us) server_negotiation.cc:299] Negotiation successful
0925 04:44:44.715840 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 04:44:44.716006 7483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:44.701526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45632 (local address 127.0.98.124:41887)
0925 04:44:44.701641 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:44.701646 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:44.702455 (+ 809us) server_negotiation.cc:407] Connection header received
0925 04:44:44.702865 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:44.702868 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:44.702935 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:44.702988 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:44.707472 (+ 4484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:44.708275 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:44.709294 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:44.709710 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:44.709818 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:44.711675 (+ 1857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:44.711679 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:44.711682 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:44.711710 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:44.714002 (+ 2292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:44.714713 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:44.714717 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:44.714719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:44.714799 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:44.715114 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:44.715118 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:44.715120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:44.715305 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:44.715393 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:44.715685 (+ 292us) server_negotiation.cc:299] Negotiation successful
0925 04:44:44.715815 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":50,"spinlock_wait_cycles":9216}
I20260925 04:44:45.666761 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:45.666107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:46181 (local address 127.0.98.65:36525)
0925 04:44:45.666503 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:45.666620 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":87,"threads_started":1}
W20260925 04:44:45.667089 7479 heartbeater.cc:644] Failed to heartbeat to 127.0.98.126:46181 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.126:46181: connect: Connection refused (error 111)
I20260925 04:44:45.671204 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:45.670471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:46181 (local address 127.0.98.67:53947)
0925 04:44:45.670973 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:45.671074 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":79,"threads_started":1}
W20260925 04:44:45.671597 7752 heartbeater.cc:644] Failed to heartbeat to 127.0.98.126:46181 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.126:46181: connect: Connection refused (error 111)
I20260925 04:44:45.671798 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:45.670710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:46181 (local address 127.0.98.66:42435)
0925 04:44:45.671596 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:45.671694 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":793,"thread_start_us":65,"threads_started":1}
W20260925 04:44:45.672048 7616 heartbeater.cc:644] Failed to heartbeat to 127.0.98.126:46181 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.126:46181: connect: Connection refused (error 111)
I20260925 04:44:46.181313 7822 raft_consensus.cc:482] T 00000000000000000000000000000000 P b655e5719e73450cb120a45e38394900 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58edb7db5329430685f0d9e7a5098394)
I20260925 04:44:46.181448 7822 raft_consensus.cc:504] T 00000000000000000000000000000000 P b655e5719e73450cb120a45e38394900 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "58edb7db5329430685f0d9e7a5098394" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46181 } } peers { permanent_uuid: "4f99000937494cc99e7b9bd62901f3f0" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 45861 } attrs { promote: false } } peers { permanent_uuid: "b655e5719e73450cb120a45e38394900" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 41887 } attrs { promote: false } }
I20260925 04:44:46.181931 7822 leader_election.cc:288] T 00000000000000000000000000000000 P b655e5719e73450cb120a45e38394900 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58edb7db5329430685f0d9e7a5098394 (127.0.98.126:46181), 4f99000937494cc99e7b9bd62901f3f0 (127.0.98.125:45861)
I20260925 04:44:46.182387 7823 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4f99000937494cc99e7b9bd62901f3f0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58edb7db5329430685f0d9e7a5098394)
I20260925 04:44:46.182674 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:46.182107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:46181 (local address 127.0.0.1:39908)
0925 04:44:46.182452 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:46.182568 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":73,"threads_started":1}
I20260925 04:44:46.182648 7823 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4f99000937494cc99e7b9bd62901f3f0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "58edb7db5329430685f0d9e7a5098394" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46181 } } peers { permanent_uuid: "4f99000937494cc99e7b9bd62901f3f0" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 45861 } attrs { promote: false } } peers { permanent_uuid: "b655e5719e73450cb120a45e38394900" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 41887 } attrs { promote: false } }
W20260925 04:44:46.182955 7283 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.126:46181: connect: Connection refused (error 111)
I20260925 04:44:46.183858 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:46.183593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:46181 (local address 127.0.0.1:39914)
0925 04:44:46.183727 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:46.183787 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260925 04:44:46.184001 7823 leader_election.cc:288] T 00000000000000000000000000000000 P 4f99000937494cc99e7b9bd62901f3f0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58edb7db5329430685f0d9e7a5098394 (127.0.98.126:46181), b655e5719e73450cb120a45e38394900 (127.0.98.124:41887)
W20260925 04:44:46.184281 7283 leader_election.cc:334] T 00000000000000000000000000000000 P b655e5719e73450cb120a45e38394900 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 58edb7db5329430685f0d9e7a5098394 (127.0.98.126:46181): Network error: Client connection negotiation failed: client connection to 127.0.98.126:46181: connect: Connection refused (error 111)
I20260925 04:44:46.184721 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:46.184067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:46181 (local address 127.0.0.1:39928)
0925 04:44:46.184502 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:46.184584 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":65,"threads_started":1}
W20260925 04:44:46.184931 7100 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.126:46181: connect: Connection refused (error 111)
I20260925 04:44:46.185880 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:46.185626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:46181 (local address 127.0.0.1:39938)
0925 04:44:46.185762 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:46.185817 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260925 04:44:46.186712 7100 leader_election.cc:334] T 00000000000000000000000000000000 P 4f99000937494cc99e7b9bd62901f3f0 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 58edb7db5329430685f0d9e7a5098394 (127.0.98.126:46181): Network error: Client connection negotiation failed: client connection to 127.0.98.126:46181: connect: Connection refused (error 111)
I20260925 04:44:46.187055 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:46.182205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:45861 (local address 127.0.0.1:49488)
0925 04:44:46.182482 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:46.182498 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:46.182603 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:46.182960 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:46.182964 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:46.182989 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:44:46.183100 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:46.183107 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:46.184299 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:46.184304 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:46.185641 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:46.185672 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:46.186632 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:46.186636 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:46.186732 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:46.186759 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 04:44:46.186838 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 04:44:46.186911 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":27,"thread_start_us":76,"threads_started":1}
I20260925 04:44:46.187399 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:46.182294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49488 (local address 127.0.98.125:45861)
0925 04:44:46.182566 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:46.182570 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:46.182585 (+ 15us) server_negotiation.cc:407] Connection header received
0925 04:44:46.182737 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:46.182742 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:46.182791 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:46.182872 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:44:46.183235 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:46.184166 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:46.185768 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:46.186511 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:46.186603 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:46.186655 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:46.187217 (+ 562us) server_negotiation.cc:299] Negotiation successful
0925 04:44:46.187292 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":53,"threads_started":1}
I20260925 04:44:46.187811 7126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b655e5719e73450cb120a45e38394900" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "4f99000937494cc99e7b9bd62901f3f0" is_pre_election: true
I20260925 04:44:46.187971 7126 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4f99000937494cc99e7b9bd62901f3f0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b655e5719e73450cb120a45e38394900 in term 1.
I20260925 04:44:46.188369 7284 leader_election.cc:302] T 00000000000000000000000000000000 P b655e5719e73450cb120a45e38394900 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4f99000937494cc99e7b9bd62901f3f0, b655e5719e73450cb120a45e38394900; no voters: 58edb7db5329430685f0d9e7a5098394
I20260925 04:44:46.188650 7822 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b655e5719e73450cb120a45e38394900 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 04:44:46.188671 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:46.184190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:41887 (local address 127.0.0.1:45636)
0925 04:44:46.184501 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:46.184515 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:46.184600 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:46.184928 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:46.184930 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:46.184947 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:44:46.185043 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:46.185048 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:46.186168 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:46.186171 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:46.187397 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:46.187408 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:46.188258 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:46.188264 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:46.188360 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:46.188389 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 04:44:46.188450 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 04:44:46.188542 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":69,"threads_started":1}
I20260925 04:44:46.188706 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:46.184320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45636 (local address 127.0.98.124:41887)
0925 04:44:46.184576 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:46.184585 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:46.184601 (+ 16us) server_negotiation.cc:407] Connection header received
0925 04:44:46.184716 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:46.184720 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:46.184762 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:46.184837 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:44:46.185176 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:46.186058 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:46.187574 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:46.188128 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:46.188238 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:46.188326 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:46.188486 (+ 160us) server_negotiation.cc:299] Negotiation successful
0925 04:44:46.188557 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":189,"thread_start_us":98,"threads_started":1}
I20260925 04:44:46.188722 7822 raft_consensus.cc:482] T 00000000000000000000000000000000 P b655e5719e73450cb120a45e38394900 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 58edb7db5329430685f0d9e7a5098394)
I20260925 04:44:46.188773 7822 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b655e5719e73450cb120a45e38394900 [term 1 FOLLOWER]: Advancing to term 2
I20260925 04:44:46.189248 7309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4f99000937494cc99e7b9bd62901f3f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b655e5719e73450cb120a45e38394900" is_pre_election: true
I20260925 04:44:46.189888 7822 raft_consensus.cc:504] T 00000000000000000000000000000000 P b655e5719e73450cb120a45e38394900 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "58edb7db5329430685f0d9e7a5098394" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46181 } } peers { permanent_uuid: "4f99000937494cc99e7b9bd62901f3f0" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 45861 } attrs { promote: false } } peers { permanent_uuid: "b655e5719e73450cb120a45e38394900" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 41887 } attrs { promote: false } }
I20260925 04:44:46.190171 7309 raft_consensus.cc:2379] T 00000000000000000000000000000000 P b655e5719e73450cb120a45e38394900 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4f99000937494cc99e7b9bd62901f3f0 in current term 2: Already voted for candidate b655e5719e73450cb120a45e38394900 in this term.
I20260925 04:44:46.190326 7822 leader_election.cc:288] T 00000000000000000000000000000000 P b655e5719e73450cb120a45e38394900 [CANDIDATE]: Term 2 election: Requested vote from peers 58edb7db5329430685f0d9e7a5098394 (127.0.98.126:46181), 4f99000937494cc99e7b9bd62901f3f0 (127.0.98.125:45861)
I20260925 04:44:46.190657 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:46.190430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:46181 (local address 127.0.0.1:39954)
0925 04:44:46.190531 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:46.190585 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 04:44:46.190699 7126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b655e5719e73450cb120a45e38394900" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "4f99000937494cc99e7b9bd62901f3f0"
I20260925 04:44:46.190759 7103 leader_election.cc:302] T 00000000000000000000000000000000 P 4f99000937494cc99e7b9bd62901f3f0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4f99000937494cc99e7b9bd62901f3f0; no voters: 58edb7db5329430685f0d9e7a5098394, b655e5719e73450cb120a45e38394900
I20260925 04:44:46.190824 7126 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f99000937494cc99e7b9bd62901f3f0 [term 1 FOLLOWER]: Advancing to term 2
I20260925 04:44:46.191831 7126 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4f99000937494cc99e7b9bd62901f3f0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b655e5719e73450cb120a45e38394900 in term 2.
I20260925 04:44:46.192028 7823 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4f99000937494cc99e7b9bd62901f3f0 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 04:44:46.192149 7284 leader_election.cc:302] T 00000000000000000000000000000000 P b655e5719e73450cb120a45e38394900 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f99000937494cc99e7b9bd62901f3f0, b655e5719e73450cb120a45e38394900; no voters:
I20260925 04:44:46.192345 7822 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b655e5719e73450cb120a45e38394900 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 04:44:46.192663 7822 raft_consensus.cc:686] T 00000000000000000000000000000000 P b655e5719e73450cb120a45e38394900 [term 2 LEADER]: Becoming Leader. State: Replica: b655e5719e73450cb120a45e38394900, State: Running, Role: LEADER
I20260925 04:44:46.193011 7822 consensus_queue.cc:237] T 00000000000000000000000000000000 P b655e5719e73450cb120a45e38394900 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "58edb7db5329430685f0d9e7a5098394" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46181 } } peers { permanent_uuid: "4f99000937494cc99e7b9bd62901f3f0" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 45861 } attrs { promote: false } } peers { permanent_uuid: "b655e5719e73450cb120a45e38394900" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 41887 } attrs { promote: false } }
I20260925 04:44:46.193360 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:46.193075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:46181 (local address 127.0.0.1:39958)
0925 04:44:46.193245 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:46.193307 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260925 04:44:46.193535 7283 leader_election.cc:334] T 00000000000000000000000000000000 P b655e5719e73450cb120a45e38394900 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 58edb7db5329430685f0d9e7a5098394 (127.0.98.126:46181): Network error: Client connection negotiation failed: client connection to 127.0.98.126:46181: connect: Connection refused (error 111)
I20260925 04:44:46.194186 7832 sys_catalog.cc:455] T 00000000000000000000000000000000 P b655e5719e73450cb120a45e38394900 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b655e5719e73450cb120a45e38394900. Latest consensus state: current_term: 2 leader_uuid: "b655e5719e73450cb120a45e38394900" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "58edb7db5329430685f0d9e7a5098394" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46181 } } peers { permanent_uuid: "4f99000937494cc99e7b9bd62901f3f0" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 45861 } attrs { promote: false } } peers { permanent_uuid: "b655e5719e73450cb120a45e38394900" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 41887 } attrs { promote: false } } }
I20260925 04:44:46.194351 7832 sys_catalog.cc:458] T 00000000000000000000000000000000 P b655e5719e73450cb120a45e38394900 [sys.catalog]: This master's current role is: LEADER
I20260925 04:44:46.194695 7834 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:44:46.196998 7834 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:44:46.197433 7834 catalog_manager.cc:1259] Loaded cluster ID: 279c26e79db84d6982461caf8787f885
I20260925 04:44:46.197490 7834 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:44:46.198144 7834 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:44:46.198576 7834 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b655e5719e73450cb120a45e38394900: Loaded TSK: 0
I20260925 04:44:46.199393 7834 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:44:46.246985 7299 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45632:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:44:46.250168 7299 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:44:46.253638 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:46.253335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:46181 (local address 127.0.0.1:39978)
0925 04:44:46.253459 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:46.253555 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 04:44:46.253783 7126 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4f99000937494cc99e7b9bd62901f3f0 [term 2 FOLLOWER]: Refusing update from remote peer b655e5719e73450cb120a45e38394900: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260925 04:44:46.254499 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:46.254196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:46181 (local address 127.0.0.1:39984)
0925 04:44:46.254334 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:46.254395 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260925 04:44:46.254465 7832 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b655e5719e73450cb120a45e38394900 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f99000937494cc99e7b9bd62901f3f0" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 45861 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260925 04:44:46.254868 7283 consensus_peers.cc:487] T 00000000000000000000000000000000 P b655e5719e73450cb120a45e38394900 -> Peer 58edb7db5329430685f0d9e7a5098394 (127.0.98.126:46181): Couldn't send request to peer 58edb7db5329430685f0d9e7a5098394. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.126:46181: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 04:44:46.256042 7823 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f99000937494cc99e7b9bd62901f3f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b655e5719e73450cb120a45e38394900. Latest consensus state: current_term: 2 leader_uuid: "b655e5719e73450cb120a45e38394900" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "58edb7db5329430685f0d9e7a5098394" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46181 } } peers { permanent_uuid: "4f99000937494cc99e7b9bd62901f3f0" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 45861 } attrs { promote: false } } peers { permanent_uuid: "b655e5719e73450cb120a45e38394900" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 41887 } attrs { promote: false } } }
I20260925 04:44:46.256188 7823 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f99000937494cc99e7b9bd62901f3f0 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:44:46.257354 7832 sys_catalog.cc:455] T 00000000000000000000000000000000 P b655e5719e73450cb120a45e38394900 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "b655e5719e73450cb120a45e38394900" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "58edb7db5329430685f0d9e7a5098394" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46181 } } peers { permanent_uuid: "4f99000937494cc99e7b9bd62901f3f0" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 45861 } attrs { promote: false } } peers { permanent_uuid: "b655e5719e73450cb120a45e38394900" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 41887 } attrs { promote: false } } }
I20260925 04:44:46.257480 7832 sys_catalog.cc:458] T 00000000000000000000000000000000 P b655e5719e73450cb120a45e38394900 [sys.catalog]: This master's current role is: LEADER
I20260925 04:44:46.258285 7836 mvcc.cc:205] Tried to move back new op lower bound from 7333115847689121792 to 7333115847449468928. Current Snapshot: MvccSnapshot[applied={T|T < 7333115847689121792}]
I20260925 04:44:46.260330 7823 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f99000937494cc99e7b9bd62901f3f0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "b655e5719e73450cb120a45e38394900" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "58edb7db5329430685f0d9e7a5098394" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46181 } } peers { permanent_uuid: "4f99000937494cc99e7b9bd62901f3f0" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 45861 } attrs { promote: false } } peers { permanent_uuid: "b655e5719e73450cb120a45e38394900" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 41887 } attrs { promote: false } } }
I20260925 04:44:46.260452 7823 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f99000937494cc99e7b9bd62901f3f0 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:44:46.268029 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:46.263604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:36667 (local address 127.0.0.1:41136)
0925 04:44:46.263817 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:46.263831 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:46.263900 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:46.264536 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:46.264539 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:46.264557 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:44:46.264664 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:46.264674 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:46.265517 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:46.265523 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:46.266693 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:46.266702 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:46.267661 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:46.267664 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:46.267751 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:46.267773 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 04:44:46.267836 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 04:44:46.267904 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145}
I20260925 04:44:46.268416 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:46.263815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41136 (local address 127.0.98.66:36667)
0925 04:44:46.264274 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:46.264279 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:46.264293 (+ 14us) server_negotiation.cc:407] Connection header received
0925 04:44:46.264343 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:46.264345 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:46.264402 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:46.264803 (+ 401us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:44:46.264893 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:46.265405 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:46.266828 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:46.267443 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:46.267543 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:46.267608 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:46.268194 (+ 586us) server_negotiation.cc:299] Negotiation successful
0925 04:44:46.268286 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":77,"threads_started":1}
I20260925 04:44:46.268953 7549 tablet_service.cc:1467] Processing CreateTablet for tablet 15c0ba84d76e458b93b995bc06c3a3d5 (DEFAULT_TABLE table=test-table [id=479f8f0ad5474f4d922d5c76d355f980]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:44:46.269273 7549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15c0ba84d76e458b93b995bc06c3a3d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:44:46.269285 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:46.264139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:38689 (local address 127.0.0.1:43236)
0925 04:44:46.265384 (+ 1245us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:46.265398 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:46.265481 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:46.265820 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:46.265823 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:46.265838 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:44:46.265931 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:46.265936 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:46.267213 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:46.267219 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:46.267950 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:46.267958 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:46.268885 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:46.268889 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:46.268979 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:46.269013 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 04:44:46.269081 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 04:44:46.269175 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1180,"mutex_wait_us":173,"thread_start_us":114,"threads_started":1}
I20260925 04:44:46.269495 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:46.264139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:46321 (local address 127.0.0.1:46878)
0925 04:44:46.264560 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:46.264573 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:46.264656 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:46.265886 (+ 1230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:46.265889 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:46.265906 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:44:46.266011 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:46.266017 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:46.267107 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:46.267111 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:46.268198 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:46.268207 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:46.269087 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:46.269090 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:46.269220 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:46.269248 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 04:44:46.269326 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 04:44:46.269391 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"mutex_wait_us":22}
I20260925 04:44:46.269601 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:46.264607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46878 (local address 127.0.98.65:46321)
0925 04:44:46.265655 (+ 1048us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:46.265659 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:46.265672 (+ 13us) server_negotiation.cc:407] Connection header received
0925 04:44:46.265718 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:46.265721 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:46.265764 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:46.266117 (+ 353us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:44:46.266229 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:46.266993 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:46.268357 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:46.268968 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:46.269083 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:46.269144 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:46.269370 (+ 226us) server_negotiation.cc:299] Negotiation successful
0925 04:44:46.269455 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":971,"thread_start_us":72,"threads_started":1}
I20260925 04:44:46.269771 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:46.265370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43236 (local address 127.0.98.67:38689)
0925 04:44:46.265614 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:46.265618 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:46.265633 (+ 15us) server_negotiation.cc:407] Connection header received
0925 04:44:46.265682 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:46.265684 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:46.265719 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:46.265815 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:44:46.266042 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:46.267125 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:46.268067 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:46.268761 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:46.268867 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:46.268926 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:46.269551 (+ 625us) server_negotiation.cc:299] Negotiation successful
0925 04:44:46.269636 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":76,"threads_started":1}
I20260925 04:44:46.270079 7412 tablet_service.cc:1467] Processing CreateTablet for tablet 15c0ba84d76e458b93b995bc06c3a3d5 (DEFAULT_TABLE table=test-table [id=479f8f0ad5474f4d922d5c76d355f980]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:44:46.270282 7685 tablet_service.cc:1467] Processing CreateTablet for tablet 15c0ba84d76e458b93b995bc06c3a3d5 (DEFAULT_TABLE table=test-table [id=479f8f0ad5474f4d922d5c76d355f980]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:44:46.270349 7412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15c0ba84d76e458b93b995bc06c3a3d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:44:46.270535 7685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15c0ba84d76e458b93b995bc06c3a3d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:44:46.272486 7844 tablet_bootstrap.cc:492] T 15c0ba84d76e458b93b995bc06c3a3d5 P 49d951568c0a4ae2a35eeedd7daf1646: Bootstrap starting.
I20260925 04:44:46.273123 7846 tablet_bootstrap.cc:492] T 15c0ba84d76e458b93b995bc06c3a3d5 P 1870b79591534da9a311af3678fc937e: Bootstrap starting.
I20260925 04:44:46.273581 7844 tablet_bootstrap.cc:654] T 15c0ba84d76e458b93b995bc06c3a3d5 P 49d951568c0a4ae2a35eeedd7daf1646: Neither blocks nor log segments found. Creating new log.
I20260925 04:44:46.274053 7846 tablet_bootstrap.cc:654] T 15c0ba84d76e458b93b995bc06c3a3d5 P 1870b79591534da9a311af3678fc937e: Neither blocks nor log segments found. Creating new log.
I20260925 04:44:46.274464 7845 tablet_bootstrap.cc:492] T 15c0ba84d76e458b93b995bc06c3a3d5 P 0bbea834c4b244998ef191efd5bb4173: Bootstrap starting.
I20260925 04:44:46.274952 7844 tablet_bootstrap.cc:492] T 15c0ba84d76e458b93b995bc06c3a3d5 P 49d951568c0a4ae2a35eeedd7daf1646: No bootstrap required, opened a new log
I20260925 04:44:46.275048 7844 ts_tablet_manager.cc:1397] T 15c0ba84d76e458b93b995bc06c3a3d5 P 49d951568c0a4ae2a35eeedd7daf1646: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 04:44:46.275280 7845 tablet_bootstrap.cc:654] T 15c0ba84d76e458b93b995bc06c3a3d5 P 0bbea834c4b244998ef191efd5bb4173: Neither blocks nor log segments found. Creating new log.
I20260925 04:44:46.275311 7846 tablet_bootstrap.cc:492] T 15c0ba84d76e458b93b995bc06c3a3d5 P 1870b79591534da9a311af3678fc937e: No bootstrap required, opened a new log
I20260925 04:44:46.275386 7846 ts_tablet_manager.cc:1397] T 15c0ba84d76e458b93b995bc06c3a3d5 P 1870b79591534da9a311af3678fc937e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 04:44:46.275496 7844 raft_consensus.cc:348] T 15c0ba84d76e458b93b995bc06c3a3d5 P 49d951568c0a4ae2a35eeedd7daf1646 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49d951568c0a4ae2a35eeedd7daf1646" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 36667 } } peers { permanent_uuid: "1870b79591534da9a311af3678fc937e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38689 } } peers { permanent_uuid: "0bbea834c4b244998ef191efd5bb4173" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46321 } }
I20260925 04:44:46.275604 7844 raft_consensus.cc:374] T 15c0ba84d76e458b93b995bc06c3a3d5 P 49d951568c0a4ae2a35eeedd7daf1646 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:44:46.275637 7844 raft_consensus.cc:729] T 15c0ba84d76e458b93b995bc06c3a3d5 P 49d951568c0a4ae2a35eeedd7daf1646 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49d951568c0a4ae2a35eeedd7daf1646, State: Initialized, Role: FOLLOWER
I20260925 04:44:46.275761 7844 consensus_queue.cc:260] T 15c0ba84d76e458b93b995bc06c3a3d5 P 49d951568c0a4ae2a35eeedd7daf1646 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49d951568c0a4ae2a35eeedd7daf1646" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 36667 } } peers { permanent_uuid: "1870b79591534da9a311af3678fc937e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38689 } } peers { permanent_uuid: "0bbea834c4b244998ef191efd5bb4173" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46321 } }
I20260925 04:44:46.275799 7846 raft_consensus.cc:348] T 15c0ba84d76e458b93b995bc06c3a3d5 P 1870b79591534da9a311af3678fc937e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49d951568c0a4ae2a35eeedd7daf1646" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 36667 } } peers { permanent_uuid: "1870b79591534da9a311af3678fc937e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38689 } } peers { permanent_uuid: "0bbea834c4b244998ef191efd5bb4173" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46321 } }
I20260925 04:44:46.275887 7846 raft_consensus.cc:374] T 15c0ba84d76e458b93b995bc06c3a3d5 P 1870b79591534da9a311af3678fc937e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:44:46.275928 7846 raft_consensus.cc:729] T 15c0ba84d76e458b93b995bc06c3a3d5 P 1870b79591534da9a311af3678fc937e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1870b79591534da9a311af3678fc937e, State: Initialized, Role: FOLLOWER
I20260925 04:44:46.276106 7844 ts_tablet_manager.cc:1428] T 15c0ba84d76e458b93b995bc06c3a3d5 P 49d951568c0a4ae2a35eeedd7daf1646: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 04:44:46.276057 7846 consensus_queue.cc:260] T 15c0ba84d76e458b93b995bc06c3a3d5 P 1870b79591534da9a311af3678fc937e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49d951568c0a4ae2a35eeedd7daf1646" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 36667 } } peers { permanent_uuid: "1870b79591534da9a311af3678fc937e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38689 } } peers { permanent_uuid: "0bbea834c4b244998ef191efd5bb4173" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46321 } }
I20260925 04:44:46.276381 7846 ts_tablet_manager.cc:1428] T 15c0ba84d76e458b93b995bc06c3a3d5 P 1870b79591534da9a311af3678fc937e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 04:44:46.276396 7845 tablet_bootstrap.cc:492] T 15c0ba84d76e458b93b995bc06c3a3d5 P 0bbea834c4b244998ef191efd5bb4173: No bootstrap required, opened a new log
I20260925 04:44:46.276901 7845 ts_tablet_manager.cc:1397] T 15c0ba84d76e458b93b995bc06c3a3d5 P 0bbea834c4b244998ef191efd5bb4173: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 04:44:46.277412 7845 raft_consensus.cc:348] T 15c0ba84d76e458b93b995bc06c3a3d5 P 0bbea834c4b244998ef191efd5bb4173 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49d951568c0a4ae2a35eeedd7daf1646" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 36667 } } peers { permanent_uuid: "1870b79591534da9a311af3678fc937e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38689 } } peers { permanent_uuid: "0bbea834c4b244998ef191efd5bb4173" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46321 } }
I20260925 04:44:46.277520 7845 raft_consensus.cc:374] T 15c0ba84d76e458b93b995bc06c3a3d5 P 0bbea834c4b244998ef191efd5bb4173 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:44:46.277554 7845 raft_consensus.cc:729] T 15c0ba84d76e458b93b995bc06c3a3d5 P 0bbea834c4b244998ef191efd5bb4173 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bbea834c4b244998ef191efd5bb4173, State: Initialized, Role: FOLLOWER
I20260925 04:44:46.277730 7845 consensus_queue.cc:260] T 15c0ba84d76e458b93b995bc06c3a3d5 P 0bbea834c4b244998ef191efd5bb4173 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49d951568c0a4ae2a35eeedd7daf1646" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 36667 } } peers { permanent_uuid: "1870b79591534da9a311af3678fc937e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38689 } } peers { permanent_uuid: "0bbea834c4b244998ef191efd5bb4173" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46321 } }
I20260925 04:44:46.277945 7853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:46.277523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:46181 (local address 127.0.98.66:58707)
0925 04:44:46.277784 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:46.277847 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":95,"threads_started":1}
I20260925 04:44:46.278059 7845 ts_tablet_manager.cc:1428] T 15c0ba84d76e458b93b995bc06c3a3d5 P 0bbea834c4b244998ef191efd5bb4173: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 04:44:46.279047 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:46.277176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:46181 (local address 127.0.98.67:48209)
0925 04:44:46.278860 (+ 1684us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:46.278929 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1615,"thread_start_us":88,"threads_started":1}
I20260925 04:44:46.279951 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:46.279704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:46181 (local address 127.0.98.67:56273)
0925 04:44:46.279836 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:46.279889 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260925 04:44:46.280138 7751 heartbeater.cc:503] Master 127.0.98.124:41887 was elected leader, sending a full tablet report...
I20260925 04:44:46.280697 7478 heartbeater.cc:503] Master 127.0.98.124:41887 was elected leader, sending a full tablet report...
I20260925 04:44:46.281436 7615 heartbeater.cc:503] Master 127.0.98.124:41887 was elected leader, sending a full tablet report...
I20260925 04:44:46.282231 7853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:46.281978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:46181 (local address 127.0.98.66:44313)
0925 04:44:46.282109 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:46.282164 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 04:44:46.282552 7855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:46.278882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:46181 (local address 127.0.98.65:40665)
0925 04:44:46.282231 (+ 3349us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:46.282312 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3279,"thread_start_us":101,"threads_started":1}
I20260925 04:44:46.283248 7855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:46.283032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:46181 (local address 127.0.98.65:55933)
0925 04:44:46.283142 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:46.283189 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 04:44:46.467819 7754 tablet.cc:2369] T 15c0ba84d76e458b93b995bc06c3a3d5 P 1870b79591534da9a311af3678fc937e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:44:46.491896 7850 raft_consensus.cc:482] T 15c0ba84d76e458b93b995bc06c3a3d5 P 49d951568c0a4ae2a35eeedd7daf1646 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:44:46.492046 7850 raft_consensus.cc:504] T 15c0ba84d76e458b93b995bc06c3a3d5 P 49d951568c0a4ae2a35eeedd7daf1646 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49d951568c0a4ae2a35eeedd7daf1646" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 36667 } } peers { permanent_uuid: "1870b79591534da9a311af3678fc937e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38689 } } peers { permanent_uuid: "0bbea834c4b244998ef191efd5bb4173" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46321 } }
I20260925 04:44:46.492563 7850 leader_election.cc:288] T 15c0ba84d76e458b93b995bc06c3a3d5 P 49d951568c0a4ae2a35eeedd7daf1646 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1870b79591534da9a311af3678fc937e (127.0.98.67:38689), 0bbea834c4b244998ef191efd5bb4173 (127.0.98.65:46321)
I20260925 04:44:46.493033 7705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15c0ba84d76e458b93b995bc06c3a3d5" candidate_uuid: "49d951568c0a4ae2a35eeedd7daf1646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1870b79591534da9a311af3678fc937e" is_pre_election: true
I20260925 04:44:46.493113 7432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15c0ba84d76e458b93b995bc06c3a3d5" candidate_uuid: "49d951568c0a4ae2a35eeedd7daf1646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bbea834c4b244998ef191efd5bb4173" is_pre_election: true
I20260925 04:44:46.493211 7705 raft_consensus.cc:2454] T 15c0ba84d76e458b93b995bc06c3a3d5 P 1870b79591534da9a311af3678fc937e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49d951568c0a4ae2a35eeedd7daf1646 in term 0.
I20260925 04:44:46.493301 7432 raft_consensus.cc:2454] T 15c0ba84d76e458b93b995bc06c3a3d5 P 0bbea834c4b244998ef191efd5bb4173 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49d951568c0a4ae2a35eeedd7daf1646 in term 0.
I20260925 04:44:46.493682 7502 leader_election.cc:302] T 15c0ba84d76e458b93b995bc06c3a3d5 P 49d951568c0a4ae2a35eeedd7daf1646 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1870b79591534da9a311af3678fc937e, 49d951568c0a4ae2a35eeedd7daf1646; no voters:
I20260925 04:44:46.493870 7850 raft_consensus.cc:2781] T 15c0ba84d76e458b93b995bc06c3a3d5 P 49d951568c0a4ae2a35eeedd7daf1646 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:44:46.493945 7850 raft_consensus.cc:482] T 15c0ba84d76e458b93b995bc06c3a3d5 P 49d951568c0a4ae2a35eeedd7daf1646 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:44:46.494001 7850 raft_consensus.cc:3037] T 15c0ba84d76e458b93b995bc06c3a3d5 P 49d951568c0a4ae2a35eeedd7daf1646 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:44:46.495117 7850 raft_consensus.cc:504] T 15c0ba84d76e458b93b995bc06c3a3d5 P 49d951568c0a4ae2a35eeedd7daf1646 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49d951568c0a4ae2a35eeedd7daf1646" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 36667 } } peers { permanent_uuid: "1870b79591534da9a311af3678fc937e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38689 } } peers { permanent_uuid: "0bbea834c4b244998ef191efd5bb4173" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46321 } }
I20260925 04:44:46.495473 7850 leader_election.cc:288] T 15c0ba84d76e458b93b995bc06c3a3d5 P 49d951568c0a4ae2a35eeedd7daf1646 [CANDIDATE]: Term 1 election: Requested vote from peers 1870b79591534da9a311af3678fc937e (127.0.98.67:38689), 0bbea834c4b244998ef191efd5bb4173 (127.0.98.65:46321)
I20260925 04:44:46.495860 7705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15c0ba84d76e458b93b995bc06c3a3d5" candidate_uuid: "49d951568c0a4ae2a35eeedd7daf1646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1870b79591534da9a311af3678fc937e"
I20260925 04:44:46.495854 7432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15c0ba84d76e458b93b995bc06c3a3d5" candidate_uuid: "49d951568c0a4ae2a35eeedd7daf1646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bbea834c4b244998ef191efd5bb4173"
I20260925 04:44:46.495982 7705 raft_consensus.cc:3037] T 15c0ba84d76e458b93b995bc06c3a3d5 P 1870b79591534da9a311af3678fc937e [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:44:46.495992 7432 raft_consensus.cc:3037] T 15c0ba84d76e458b93b995bc06c3a3d5 P 0bbea834c4b244998ef191efd5bb4173 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:44:46.497097 7705 raft_consensus.cc:2454] T 15c0ba84d76e458b93b995bc06c3a3d5 P 1870b79591534da9a311af3678fc937e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49d951568c0a4ae2a35eeedd7daf1646 in term 1.
I20260925 04:44:46.497097 7432 raft_consensus.cc:2454] T 15c0ba84d76e458b93b995bc06c3a3d5 P 0bbea834c4b244998ef191efd5bb4173 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49d951568c0a4ae2a35eeedd7daf1646 in term 1.
I20260925 04:44:46.497407 7504 leader_election.cc:302] T 15c0ba84d76e458b93b995bc06c3a3d5 P 49d951568c0a4ae2a35eeedd7daf1646 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bbea834c4b244998ef191efd5bb4173, 49d951568c0a4ae2a35eeedd7daf1646; no voters:
I20260925 04:44:46.497617 7850 raft_consensus.cc:2781] T 15c0ba84d76e458b93b995bc06c3a3d5 P 49d951568c0a4ae2a35eeedd7daf1646 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:44:46.497859 7850 raft_consensus.cc:686] T 15c0ba84d76e458b93b995bc06c3a3d5 P 49d951568c0a4ae2a35eeedd7daf1646 [term 1 LEADER]: Becoming Leader. State: Replica: 49d951568c0a4ae2a35eeedd7daf1646, State: Running, Role: LEADER
I20260925 04:44:46.498315 7850 consensus_queue.cc:237] T 15c0ba84d76e458b93b995bc06c3a3d5 P 49d951568c0a4ae2a35eeedd7daf1646 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49d951568c0a4ae2a35eeedd7daf1646" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 36667 } } peers { permanent_uuid: "1870b79591534da9a311af3678fc937e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38689 } } peers { permanent_uuid: "0bbea834c4b244998ef191efd5bb4173" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46321 } }
I20260925 04:44:46.498723 7853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:46.498392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:46181 (local address 127.0.98.66:43793)
0925 04:44:46.498555 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:46.498642 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 04:44:46.500409 7297 catalog_manager.cc:5495] T 15c0ba84d76e458b93b995bc06c3a3d5 P 49d951568c0a4ae2a35eeedd7daf1646 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49d951568c0a4ae2a35eeedd7daf1646 (127.0.98.66). New cstate: current_term: 1 leader_uuid: "49d951568c0a4ae2a35eeedd7daf1646" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49d951568c0a4ae2a35eeedd7daf1646" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 36667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1870b79591534da9a311af3678fc937e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0bbea834c4b244998ef191efd5bb4173" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46321 } health_report { overall_health: UNKNOWN } } }
W20260925 04:44:46.516845 7618 tablet.cc:2369] T 15c0ba84d76e458b93b995bc06c3a3d5 P 49d951568c0a4ae2a35eeedd7daf1646: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:44:46.566955 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:46.563567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41146 (local address 127.0.98.66:36667)
0925 04:44:46.563735 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:46.563738 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:46.563753 (+ 15us) server_negotiation.cc:407] Connection header received
0925 04:44:46.563806 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:46.563808 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:46.563857 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:46.563932 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:44:46.564284 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:46.564845 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:46.565775 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:46.566014 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:46.566109 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:46.566445 (+ 336us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:44:46.566537 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:44:46.566625 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:46.566747 (+ 122us) server_negotiation.cc:299] Negotiation successful
0925 04:44:46.566805 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260925 04:44:46.569376 7705 raft_consensus.cc:1261] T 15c0ba84d76e458b93b995bc06c3a3d5 P 1870b79591534da9a311af3678fc937e [term 1 FOLLOWER]: Refusing update from remote peer 49d951568c0a4ae2a35eeedd7daf1646: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:44:46.569376 7432 raft_consensus.cc:1261] T 15c0ba84d76e458b93b995bc06c3a3d5 P 0bbea834c4b244998ef191efd5bb4173 [term 1 FOLLOWER]: Refusing update from remote peer 49d951568c0a4ae2a35eeedd7daf1646: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:44:46.569859 7850 consensus_queue.cc:1035] T 15c0ba84d76e458b93b995bc06c3a3d5 P 49d951568c0a4ae2a35eeedd7daf1646 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bbea834c4b244998ef191efd5bb4173" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:44:46.570045 7856 consensus_queue.cc:1035] T 15c0ba84d76e458b93b995bc06c3a3d5 P 49d951568c0a4ae2a35eeedd7daf1646 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1870b79591534da9a311af3678fc937e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:44:46.571988 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:46.571652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:46181 (local address 127.0.98.67:46865)
0925 04:44:46.571816 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:46.571881 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260925 04:44:46.572556 7855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:46.572280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:46181 (local address 127.0.98.65:56181)
0925 04:44:46.572427 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:46.572480 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260925 04:44:46.576001 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:46.575323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:46181 (local address 127.0.98.67:37443)
0925 04:44:46.575845 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:46.575905 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444}
I20260925 04:44:46.577409 7853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:46.576550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:46181 (local address 127.0.98.66:35061)
0925 04:44:46.577275 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:46.577336 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":646}
I20260925 04:44:46.577548 7855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:46.577317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:46181 (local address 127.0.98.65:55347)
0925 04:44:46.577417 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:46.577471 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 04:44:46.579523 7861 mvcc.cc:205] Tried to move back new op lower bound from 7333115848983490560 to 7333115848698843136. Current Snapshot: MvccSnapshot[applied={T|T < 7333115848983490560}]
I20260925 04:44:46.580264 7853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:46.579973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:46181 (local address 127.0.98.66:37711)
0925 04:44:46.580104 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:46.580170 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 04:44:46.581071 7299 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45632:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:44:46.581240 7299 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45632:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:44:46.585271 7299 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b655e5719e73450cb120a45e38394900: Sending DeleteTablet for 3 replicas of tablet 15c0ba84d76e458b93b995bc06c3a3d5
I20260925 04:44:46.586061 7549 tablet_service.cc:1514] Processing DeleteTablet for tablet 15c0ba84d76e458b93b995bc06c3a3d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:44:46 UTC) from {username='oryx', principal='oryx/127.0.98.124@KRBTEST.COM'} at 127.0.0.1:41136
I20260925 04:44:46.586091 7685 tablet_service.cc:1514] Processing DeleteTablet for tablet 15c0ba84d76e458b93b995bc06c3a3d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:44:46 UTC) from {username='oryx', principal='oryx/127.0.98.124@KRBTEST.COM'} at 127.0.0.1:43236
I20260925 04:44:46.586279 7412 tablet_service.cc:1514] Processing DeleteTablet for tablet 15c0ba84d76e458b93b995bc06c3a3d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:44:46 UTC) from {username='oryx', principal='oryx/127.0.98.124@KRBTEST.COM'} at 127.0.0.1:46878
I20260925 04:44:46.586553 7868 tablet_replica.cc:330] T 15c0ba84d76e458b93b995bc06c3a3d5 P 49d951568c0a4ae2a35eeedd7daf1646: stopping tablet replica
I20260925 04:44:46.586701 7868 raft_consensus.cc:2229] T 15c0ba84d76e458b93b995bc06c3a3d5 P 49d951568c0a4ae2a35eeedd7daf1646 [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:44:46.586728 7870 tablet_replica.cc:330] T 15c0ba84d76e458b93b995bc06c3a3d5 P 0bbea834c4b244998ef191efd5bb4173: stopping tablet replica
I20260925 04:44:46.586830 7870 raft_consensus.cc:2229] T 15c0ba84d76e458b93b995bc06c3a3d5 P 0bbea834c4b244998ef191efd5bb4173 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:44:46.586912 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 7260
I20260925 04:44:46.586925 7868 raft_consensus.cc:2258] T 15c0ba84d76e458b93b995bc06c3a3d5 P 49d951568c0a4ae2a35eeedd7daf1646 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:44:46.586977 7870 raft_consensus.cc:2258] T 15c0ba84d76e458b93b995bc06c3a3d5 P 0bbea834c4b244998ef191efd5bb4173 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:44:46.587208 7869 tablet_replica.cc:330] T 15c0ba84d76e458b93b995bc06c3a3d5 P 1870b79591534da9a311af3678fc937e: stopping tablet replica
I20260925 04:44:46.587347 7869 raft_consensus.cc:2229] T 15c0ba84d76e458b93b995bc06c3a3d5 P 1870b79591534da9a311af3678fc937e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:44:46.587507 7869 raft_consensus.cc:2258] T 15c0ba84d76e458b93b995bc06c3a3d5 P 1870b79591534da9a311af3678fc937e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:44:46.588274 7868 ts_tablet_manager.cc:1905] T 15c0ba84d76e458b93b995bc06c3a3d5 P 49d951568c0a4ae2a35eeedd7daf1646: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:44:46.588546 7869 ts_tablet_manager.cc:1905] T 15c0ba84d76e458b93b995bc06c3a3d5 P 1870b79591534da9a311af3678fc937e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:44:46.591229 7869 ts_tablet_manager.cc:1918] T 15c0ba84d76e458b93b995bc06c3a3d5 P 1870b79591534da9a311af3678fc937e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:44:46.591333 7869 log.cc:1196] T 15c0ba84d76e458b93b995bc06c3a3d5 P 1870b79591534da9a311af3678fc937e: Deleting WAL directory at /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-2/wal/wals/15c0ba84d76e458b93b995bc06c3a3d5
I20260925 04:44:46.591593 7869 ts_tablet_manager.cc:1939] T 15c0ba84d76e458b93b995bc06c3a3d5 P 1870b79591534da9a311af3678fc937e: Deleting consensus metadata
I20260925 04:44:46.591929 7868 ts_tablet_manager.cc:1918] T 15c0ba84d76e458b93b995bc06c3a3d5 P 49d951568c0a4ae2a35eeedd7daf1646: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:44:46.592027 7868 log.cc:1196] T 15c0ba84d76e458b93b995bc06c3a3d5 P 49d951568c0a4ae2a35eeedd7daf1646: Deleting WAL directory at /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790311449873301-393-0/minicluster-data/ts-1/wal/wals/15c0ba84d76e458b93b995bc06c3a3d5
I20260925 04:44:46.592344 7868 ts_tablet_manager.cc:1939] T 15c0ba84d76e458b93b995bc06c3a3d5 P 49d951568c0a4ae2a35eeedd7daf1646: Deleting consensus metadata
I20260925 04:44:46.592574 7283 catalog_manager.cc:4841] TS 1870b79591534da9a311af3678fc937e (127.0.98.67:38689): tablet 15c0ba84d76e458b93b995bc06c3a3d5 (table test-table [id=479f8f0ad5474f4d922d5c76d355f980]) successfully deleted
I20260925 04:44:46.593206 7284 catalog_manager.cc:4841] TS 49d951568c0a4ae2a35eeedd7daf1646 (127.0.98.66:36667): tablet 15c0ba84d76e458b93b995bc06c3a3d5 (table test-table [id=479f8f0ad5474f4d922d5c76d355f980]) successfully deleted
I20260925 04:44:46.595439 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 7486
I20260925 04:44:46.597470 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:46.597216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:46321 (local address 127.0.0.1:46882)
0925 04:44:46.597325 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:46.597388 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.65:46321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260925 04:44:46.597761 7285 catalog_manager.cc:4584] TS 0bbea834c4b244998ef191efd5bb4173 (127.0.98.65:46321): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 15c0ba84d76e458b93b995bc06c3a3d5: Network error: Client connection negotiation failed: client connection to 127.0.98.65:46321: connect: Connection refused (error 111)
I20260925 04:44:46.602104 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 7622
I20260925 04:44:46.608448 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 6980
I20260925 04:44:46.614699 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 7082
2026-09-25T04:44:46Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5528 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:44:46 dist-test-slave-kghz krb5kdc[7873](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:44:46 dist-test-slave-kghz krb5kdc[7873](info): set up 2 sockets
Sep 25 04:44:46 dist-test-slave-kghz krb5kdc[7873](info): commencing operation
krb5kdc: starting...
W20260925 04:44:47.662961 393 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/krb5kdc/test-user.keytab.
Sep 25 04:44:47 dist-test-slave-kghz krb5kdc[7873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311487, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:44:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:44:47Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/krb5kdc/oryx_127.0.98.126.keytab.
Entry for principal oryx/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/krb5kdc/oryx_127.0.98.126.keytab.
I20260925 04:44:47.793011 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:40947
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/krb5kdc/oryx_127.0.98.126.keytab
--principal=oryx/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:37893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:40947
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:44:47.909266 7886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:44:47.909556 7886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:44:47.909600 7886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:44:47.913784 7886 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:44:47.913862 7886 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 04:44:47.913880 7886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:44:47.913899 7886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:44:47.913916 7886 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:44:47.913935 7886 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:44:47.919552 7886 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:37893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:40947
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/krb5kdc/oryx_127.0.98.126.keytab
--principal=oryx/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:40947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7886
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:44:47.920603 7886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:44:47.921571 7886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 04:44:47.930617 7886 server_base.cc:1034] running on GCE node
W20260925 04:44:47.930678 7894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:47.930562 7891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:47.930562 7892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:47.931638 7886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:44:47.932775 7886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:44:47.934007 7886 hybrid_clock.cc:648] HybridClock initialized: now 1790311487933983 us; error 38 us; skew 500 ppm
Sep 25 04:44:47 dist-test-slave-kghz krb5kdc[7873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311487, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:44:47.936937 7886 init.cc:377] Logged in from keytab as oryx/127.0.98.126@KRBTEST.COM (short username oryx)
I20260925 04:44:47.937871 7886 webserver.cc:460] Webserver started at http://127.0.98.126:38999/ using document root <none> and password file <none>
I20260925 04:44:47.938516 7886 fs_manager.cc:359] Metadata directory not provided
I20260925 04:44:47.938596 7886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:44:47.938844 7886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:44:47.941066 7886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/minicluster-data/master-0/data/instance:
uuid: "1f45ce55bb3f415c8d7178965484a569"
format_stamp: "Formatted at 2026-09-25 04:44:47 on dist-test-slave-kghz"
server_key: "ce089b82d08129b96faa994749153079"
server_key_iv: "d533a9062a050579cbbeee823b328c0d"
server_key_version: "encryptionkey@0"
I20260925 04:44:47.941671 7886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/minicluster-data/master-0/wal/instance:
uuid: "1f45ce55bb3f415c8d7178965484a569"
format_stamp: "Formatted at 2026-09-25 04:44:47 on dist-test-slave-kghz"
server_key: "ce089b82d08129b96faa994749153079"
server_key_iv: "d533a9062a050579cbbeee823b328c0d"
server_key_version: "encryptionkey@0"
I20260925 04:44:47.945766 7886 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 04:44:47.948483 7901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:47.949868 7886 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 04:44:47.950024 7886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/minicluster-data/master-0/wal
uuid: "1f45ce55bb3f415c8d7178965484a569"
format_stamp: "Formatted at 2026-09-25 04:44:47 on dist-test-slave-kghz"
server_key: "ce089b82d08129b96faa994749153079"
server_key_iv: "d533a9062a050579cbbeee823b328c0d"
server_key_version: "encryptionkey@0"
I20260925 04:44:47.950152 7886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:44:47.976954 7886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:44:47.980479 7886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:44:47.980713 7886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:44:47.989172 7886 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:40947
I20260925 04:44:47.989180 7963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:40947 every 8 connection(s)
I20260925 04:44:47.990233 7886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
I20260925 04:44:47.993094 7964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:44:47.998574 7964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f45ce55bb3f415c8d7178965484a569: Bootstrap starting.
I20260925 04:44:47.999977 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 7886
I20260925 04:44:48.000116 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/minicluster-data/master-0/wal/instance
I20260925 04:44:48.000391 393 external_mini_cluster.cc:1428] Setting key e422b1a8faab03934580b36d633f1a53
I20260925 04:44:48.001473 7964 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f45ce55bb3f415c8d7178965484a569: Neither blocks nor log segments found. Creating new log.
I20260925 04:44:48.002380 7964 log.cc:824] T 00000000000000000000000000000000 P 1f45ce55bb3f415c8d7178965484a569: Log is configured to *not* fsync() on all Append() calls
I20260925 04:44:48.004675 7964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f45ce55bb3f415c8d7178965484a569: No bootstrap required, opened a new log
I20260925 04:44:48.008855 7964 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f45ce55bb3f415c8d7178965484a569 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f45ce55bb3f415c8d7178965484a569" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 40947 } }
I20260925 04:44:48.009181 7964 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1f45ce55bb3f415c8d7178965484a569 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:44:48.009253 7964 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f45ce55bb3f415c8d7178965484a569 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f45ce55bb3f415c8d7178965484a569, State: Initialized, Role: FOLLOWER
I20260925 04:44:48.009768 7964 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f45ce55bb3f415c8d7178965484a569 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f45ce55bb3f415c8d7178965484a569" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 40947 } }
I20260925 04:44:48.009903 7964 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1f45ce55bb3f415c8d7178965484a569 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:44:48.009986 7964 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f45ce55bb3f415c8d7178965484a569 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:44:48.010089 7964 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f45ce55bb3f415c8d7178965484a569 [term 0 FOLLOWER]: Advancing to term 1
Sep 25 04:44:48 dist-test-slave-kghz krb5kdc[7873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311487, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM
I20260925 04:44:48.011081 7964 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f45ce55bb3f415c8d7178965484a569 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f45ce55bb3f415c8d7178965484a569" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 40947 } }
I20260925 04:44:48.011420 7964 leader_election.cc:302] T 00000000000000000000000000000000 P 1f45ce55bb3f415c8d7178965484a569 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1f45ce55bb3f415c8d7178965484a569; no voters:
I20260925 04:44:48.011741 7964 leader_election.cc:288] T 00000000000000000000000000000000 P 1f45ce55bb3f415c8d7178965484a569 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:44:48.011894 7969 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f45ce55bb3f415c8d7178965484a569 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:44:48.012158 7969 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1f45ce55bb3f415c8d7178965484a569 [term 1 LEADER]: Becoming Leader. State: Replica: 1f45ce55bb3f415c8d7178965484a569, State: Running, Role: LEADER
I20260925 04:44:48.012599 7969 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f45ce55bb3f415c8d7178965484a569 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f45ce55bb3f415c8d7178965484a569" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 40947 } }
I20260925 04:44:48.013188 7964 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f45ce55bb3f415c8d7178965484a569 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:44:48.014456 7971 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f45ce55bb3f415c8d7178965484a569 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1f45ce55bb3f415c8d7178965484a569" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f45ce55bb3f415c8d7178965484a569" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 40947 } } }
I20260925 04:44:48.014892 7971 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f45ce55bb3f415c8d7178965484a569 [sys.catalog]: This master's current role is: LEADER
I20260925 04:44:48.014775 7970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f45ce55bb3f415c8d7178965484a569 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f45ce55bb3f415c8d7178965484a569. Latest consensus state: current_term: 1 leader_uuid: "1f45ce55bb3f415c8d7178965484a569" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f45ce55bb3f415c8d7178965484a569" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 40947 } } }
I20260925 04:44:48.015229 7970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f45ce55bb3f415c8d7178965484a569 [sys.catalog]: This master's current role is: LEADER
I20260925 04:44:48.017793 7967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:48.001632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35342 (local address 127.0.98.126:40947)
0925 04:44:48.002109 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:48.002127 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:48.002167 (+ 40us) server_negotiation.cc:407] Connection header received
0925 04:44:48.003195 (+ 1028us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:48.003222 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:48.003593 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:48.003914 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:48.004829 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:48.005963 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:48.007265 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:48.007797 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:48.007933 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:48.011081 (+ 3148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:48.011087 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:48.011101 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:48.011131 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:48.013435 (+ 2304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:48.014057 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:48.014064 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:48.014070 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:48.014228 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:48.014633 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:48.014639 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:48.014645 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:48.015187 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:48.015405 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:48.016387 (+ 982us) server_negotiation.cc:299] Negotiation successful
0925 04:44:48.016706 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":143,"threads_started":1}
W20260925 04:44:48.019866 7984 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1f45ce55bb3f415c8d7178965484a569: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 04:44:48.019999 7984 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 04:44:48.020082 7981 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:44:48.020902 7981 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:44:48.026845 7981 catalog_manager.cc:1347] Generated new cluster ID: 6a0106faa1ca4f90b4dc377b146f8caf
I20260925 04:44:48.026923 7981 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:44:48.037297 7981 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:44:48.038288 7981 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:44:48.047459 7981 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1f45ce55bb3f415c8d7178965484a569: Generated new TSK 0
I20260925 04:44:48.048194 7981 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/krb5kdc/oryx_127.0.98.65.keytab.
Entry for principal oryx/127.0.98.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/krb5kdc/oryx_127.0.98.65.keytab.
I20260925 04:44:48.081566 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:0
--local_ip_for_outbound_sockets=127.0.98.65
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:40947
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/krb5kdc/oryx_127.0.98.65.keytab
--principal=oryx/127.0.98.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:37893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:44:48.196512 7990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:44:48.196878 7990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:44:48.196965 7990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:44:48.201241 7990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:44:48.201321 7990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:44:48.201361 7990 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 04:44:48.201447 7990 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260925 04:44:48.207183 7990 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:37893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/krb5kdc/oryx_127.0.98.65.keytab
--principal=oryx/127.0.98.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.126:40947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7990
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:44:48.208302 7990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:44:48.209380 7990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 04:44:48.219820 7990 server_base.cc:1034] running on GCE node
W20260925 04:44:48.219743 7996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:48.219797 7995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:48.219743 7998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:48.220484 7990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:44:48.221190 7990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:44:48.222373 7990 hybrid_clock.cc:648] HybridClock initialized: now 1790311488222331 us; error 51 us; skew 500 ppm
Sep 25 04:44:48 dist-test-slave-kghz krb5kdc[7873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311488, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:44:48.225371 7990 init.cc:377] Logged in from keytab as oryx/127.0.98.65@KRBTEST.COM (short username oryx)
I20260925 04:44:48.226320 7990 webserver.cc:460] Webserver started at http://127.0.98.65:40065/ using document root <none> and password file <none>
I20260925 04:44:48.226903 7990 fs_manager.cc:359] Metadata directory not provided
I20260925 04:44:48.226991 7990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:44:48.227237 7990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:44:48.229300 7990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/minicluster-data/ts-0/data/instance:
uuid: "7532f3a035704530bff6b9605888f7c7"
format_stamp: "Formatted at 2026-09-25 04:44:48 on dist-test-slave-kghz"
server_key: "3429245f6f88bb9311a2b09ded312eed"
server_key_iv: "5252962b75098e584e27cc124fda067a"
server_key_version: "encryptionkey@0"
I20260925 04:44:48.229866 7990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/minicluster-data/ts-0/wal/instance:
uuid: "7532f3a035704530bff6b9605888f7c7"
format_stamp: "Formatted at 2026-09-25 04:44:48 on dist-test-slave-kghz"
server_key: "3429245f6f88bb9311a2b09ded312eed"
server_key_iv: "5252962b75098e584e27cc124fda067a"
server_key_version: "encryptionkey@0"
I20260925 04:44:48.233695 7990 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 04:44:48.236212 8005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:48.237437 7990 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 04:44:48.237691 7990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/minicluster-data/ts-0/wal
uuid: "7532f3a035704530bff6b9605888f7c7"
format_stamp: "Formatted at 2026-09-25 04:44:48 on dist-test-slave-kghz"
server_key: "3429245f6f88bb9311a2b09ded312eed"
server_key_iv: "5252962b75098e584e27cc124fda067a"
server_key_version: "encryptionkey@0"
I20260925 04:44:48.237840 7990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:44:48.257113 7990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:44:48.260504 7990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:44:48.260708 7990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:44:48.262113 7990 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:44:48.262183 7990 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:48.262274 7990 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:44:48.262313 7990 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 04:44:48 dist-test-slave-kghz krb5kdc[7873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311488, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM
I20260925 04:44:48.273630 7990 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:33301
I20260925 04:44:48.273795 8120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:33301 every 8 connection(s)
I20260925 04:44:48.274677 7990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
I20260925 04:44:48.276466 7967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:48.263114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:37025 (local address 127.0.98.126:40947)
0925 04:44:48.263268 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:48.263273 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:48.264214 (+ 941us) server_negotiation.cc:407] Connection header received
0925 04:44:48.265463 (+ 1249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:48.265468 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:48.265537 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:48.265655 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:48.267119 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:48.267779 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:48.269017 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:48.269347 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:48.269471 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:48.272459 (+ 2988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:48.272465 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:48.272469 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:48.272500 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:48.274309 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:48.274861 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:48.274865 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:48.274867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:48.274956 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:48.275435 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:48.275438 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:48.275440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:48.275672 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:48.275756 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:48.276234 (+ 478us) server_negotiation.cc:299] Negotiation successful
0925 04:44:48.276348 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 04:44:48.277273 8014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:48.263566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:40947 (local address 127.0.98.65:37025)
0925 04:44:48.264035 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:48.264086 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:48.265191 (+ 1105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:48.265920 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:48.265929 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:48.266403 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:48.266967 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:48.266981 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:48.267898 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:48.267902 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:48.268882 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:48.268892 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:48.269530 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:48.269534 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:48.269751 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:48.270407 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:48.270426 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:48.272287 (+ 1861us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:44:48.274435 (+ 2148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:48.274443 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:48.274458 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:48.274731 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:48.275118 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:48.275121 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:48.275123 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:48.275306 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:48.275776 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:44:48.275782 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:44:48.276017 (+ 235us) client_negotiation.cc:769] Sending connection context
0925 04:44:48.276181 (+ 164us) client_negotiation.cc:240] Negotiation successful
0925 04:44:48.276430 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":128,"threads_started":1}
I20260925 04:44:48.278172 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 7990
I20260925 04:44:48.278344 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/minicluster-data/ts-0/wal/instance
I20260925 04:44:48.278654 393 external_mini_cluster.cc:1428] Setting key 1e030e7545a291b93b889ab7c71b04c7
I20260925 04:44:48.278728 8121 heartbeater.cc:347] Connected to a master server at 127.0.98.126:40947
I20260925 04:44:48.278878 8121 heartbeater.cc:464] Registering TS with master...
I20260925 04:44:48.279471 8121 heartbeater.cc:511] Master 127.0.98.126:40947 requested a full tablet report, sending...
I20260925 04:44:48.281602 7917 ts_manager.cc:194] Registered new tserver with Master: 7532f3a035704530bff6b9605888f7c7 (127.0.98.65:33301)
I20260925 04:44:48.283277 7917 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.65@KRBTEST.COM'} at 127.0.98.65:37025
I20260925 04:44:48.288213 7967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:48.280916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:59685 (local address 127.0.98.126:40947)
0925 04:44:48.281030 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:48.281034 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:48.281330 (+ 296us) server_negotiation.cc:407] Connection header received
0925 04:44:48.281442 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:48.281448 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:48.281503 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:48.281589 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:48.282092 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:48.282660 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:48.283510 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:48.283802 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:48.283921 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:48.285078 (+ 1157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:48.285082 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:48.285084 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:48.285112 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:48.286643 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:48.287056 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:48.287061 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:48.287066 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:48.287146 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:48.287410 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:48.287415 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:48.287417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:48.287595 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:48.287695 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:48.287943 (+ 248us) server_negotiation.cc:299] Negotiation successful
0925 04:44:48.288053 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 04:44:48.288232 8127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:48.280960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:40947 (local address 127.0.98.65:59685)
0925 04:44:48.281235 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:48.281251 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:48.281348 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:48.281616 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:48.281622 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:48.281839 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:48.281958 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:48.281965 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:48.282785 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:48.282788 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:48.283389 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:48.283401 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:48.283914 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:48.283920 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:48.284016 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:48.284598 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:48.284611 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:48.284943 (+ 332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:44:48.286762 (+ 1819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:48.286765 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:48.286768 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:48.286964 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:48.287239 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:48.287242 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:48.287244 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:48.287326 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:48.287695 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:44:48.287699 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:44:48.287788 (+ 89us) client_negotiation.cc:769] Sending connection context
0925 04:44:48.287891 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 04:44:48.288029 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":109,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/krb5kdc/oryx_127.0.98.66.keytab.
Entry for principal oryx/127.0.98.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/krb5kdc/oryx_127.0.98.66.keytab.
I20260925 04:44:48.310792 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:0
--local_ip_for_outbound_sockets=127.0.98.66
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:40947
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/krb5kdc/oryx_127.0.98.66.keytab
--principal=oryx/127.0.98.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:37893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:44:48.425226 8130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:44:48.425571 8130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:44:48.425659 8130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:44:48.429997 8130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:44:48.430266 8130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:44:48.430397 8130 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 04:44:48.430480 8130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260925 04:44:48.436252 8130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:37893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/krb5kdc/oryx_127.0.98.66.keytab
--principal=oryx/127.0.98.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.126:40947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8130
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:44:48.437495 8130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:44:48.438652 8130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 04:44:48.449057 8130 server_base.cc:1034] running on GCE node
W20260925 04:44:48.448979 8135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:48.449002 8138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:48.448979 8136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:48.449743 8130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:44:48.450484 8130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:44:48.451675 8130 hybrid_clock.cc:648] HybridClock initialized: now 1790311488451656 us; error 31 us; skew 500 ppm
Sep 25 04:44:48 dist-test-slave-kghz krb5kdc[7873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311488, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:44:48.454674 8130 init.cc:377] Logged in from keytab as oryx/127.0.98.66@KRBTEST.COM (short username oryx)
I20260925 04:44:48.455663 8130 webserver.cc:460] Webserver started at http://127.0.98.66:43519/ using document root <none> and password file <none>
I20260925 04:44:48.456264 8130 fs_manager.cc:359] Metadata directory not provided
I20260925 04:44:48.456352 8130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:44:48.456581 8130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:44:48.458742 8130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/minicluster-data/ts-1/data/instance:
uuid: "b4886a90d7904435ada30398c846f69e"
format_stamp: "Formatted at 2026-09-25 04:44:48 on dist-test-slave-kghz"
server_key: "888b26fc4f42b977cc5c44cdb6320002"
server_key_iv: "34c98fe54d93eabd515b4f51e6328abd"
server_key_version: "encryptionkey@0"
I20260925 04:44:48.459316 8130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/minicluster-data/ts-1/wal/instance:
uuid: "b4886a90d7904435ada30398c846f69e"
format_stamp: "Formatted at 2026-09-25 04:44:48 on dist-test-slave-kghz"
server_key: "888b26fc4f42b977cc5c44cdb6320002"
server_key_iv: "34c98fe54d93eabd515b4f51e6328abd"
server_key_version: "encryptionkey@0"
I20260925 04:44:48.463289 8130 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 04:44:48.465884 8145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:48.467242 8130 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:44:48.467383 8130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/minicluster-data/ts-1/wal
uuid: "b4886a90d7904435ada30398c846f69e"
format_stamp: "Formatted at 2026-09-25 04:44:48 on dist-test-slave-kghz"
server_key: "888b26fc4f42b977cc5c44cdb6320002"
server_key_iv: "34c98fe54d93eabd515b4f51e6328abd"
server_key_version: "encryptionkey@0"
I20260925 04:44:48.467504 8130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:44:48.489452 8130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:44:48.492610 8130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:44:48.492842 8130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:44:48.494735 8130 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:44:48.494801 8130 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:48.494882 8130 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:44:48.494951 8130 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 04:44:48 dist-test-slave-kghz krb5kdc[7873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311488, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM
I20260925 04:44:48.505715 8130 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:39747
I20260925 04:44:48.505856 8260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:39747 every 8 connection(s)
I20260925 04:44:48.506901 8130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
I20260925 04:44:48.508490 7967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:48.495448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:43603 (local address 127.0.98.126:40947)
0925 04:44:48.495574 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:48.495578 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:48.496662 (+ 1084us) server_negotiation.cc:407] Connection header received
0925 04:44:48.497822 (+ 1160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:48.497827 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:48.497886 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:48.497955 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:48.499223 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:48.499818 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:48.501045 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:48.501254 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:48.501366 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:48.504453 (+ 3087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:48.504458 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:48.504462 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:48.504498 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:48.506380 (+ 1882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:48.506866 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:48.506873 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:48.506878 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:48.506972 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:48.507397 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:48.507401 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:48.507403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:48.507638 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:48.507753 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:48.508273 (+ 520us) server_negotiation.cc:299] Negotiation successful
0925 04:44:48.508374 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 04:44:48.509251 8154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:48.496015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:40947 (local address 127.0.98.66:43603)
0925 04:44:48.496495 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:48.496558 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:48.497563 (+ 1005us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:48.498171 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:48.498179 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:48.498604 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:48.499083 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:48.499097 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:48.499938 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:48.499942 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:48.500905 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:48.500915 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:48.501406 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:48.501410 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:48.501687 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:48.502416 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:48.502440 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:48.504249 (+ 1809us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:44:48.506494 (+ 2245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:48.506498 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:48.506509 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:48.506750 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:48.507089 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:48.507093 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:48.507095 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:48.507272 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:48.507748 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:44:48.507752 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:44:48.508060 (+ 308us) client_negotiation.cc:769] Sending connection context
0925 04:44:48.508217 (+ 157us) client_negotiation.cc:240] Negotiation successful
0925 04:44:48.508456 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":134,"threads_started":1}
I20260925 04:44:48.509735 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 8130
I20260925 04:44:48.509878 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/minicluster-data/ts-1/wal/instance
I20260925 04:44:48.510157 393 external_mini_cluster.cc:1428] Setting key a2a10cd66568935de6766ee79c182a28
I20260925 04:44:48.511083 8261 heartbeater.cc:347] Connected to a master server at 127.0.98.126:40947
I20260925 04:44:48.511224 8261 heartbeater.cc:464] Registering TS with master...
I20260925 04:44:48.511788 8261 heartbeater.cc:511] Master 127.0.98.126:40947 requested a full tablet report, sending...
I20260925 04:44:48.512807 7917 ts_manager.cc:194] Registered new tserver with Master: b4886a90d7904435ada30398c846f69e (127.0.98.66:39747)
I20260925 04:44:48.513367 7917 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.66@KRBTEST.COM'} at 127.0.98.66:43603
WARNING: no policy specified for oryx/127.0.98.67@KRBTEST.COM; defaulting to no policy
I20260925 04:44:48.521822 7967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:48.513609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:45537 (local address 127.0.98.126:40947)
0925 04:44:48.513749 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:48.513755 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:48.514472 (+ 717us) server_negotiation.cc:407] Connection header received
0925 04:44:48.514592 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:48.514596 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:48.514644 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:48.514734 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:48.515290 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:48.516025 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:48.516883 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:48.517126 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:48.517232 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:48.518454 (+ 1222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:48.518457 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:48.518460 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:48.518485 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:48.520162 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:48.520621 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:48.520625 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:48.520627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:48.520708 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:48.520974 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:48.520979 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:48.520980 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:48.521195 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:48.521301 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:48.521549 (+ 248us) server_negotiation.cc:299] Negotiation successful
0925 04:44:48.521653 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 04:44:48.521832 8268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:48.514109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:40947 (local address 127.0.98.66:45537)
0925 04:44:48.514383 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:48.514398 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:48.514499 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:48.514764 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:48.514766 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:48.515029 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:48.515155 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:48.515163 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:48.516203 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:48.516206 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:48.516769 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:48.516775 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:48.517235 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:48.517238 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:48.517328 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:48.517902 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:48.517914 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:48.518316 (+ 402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:44:48.520284 (+ 1968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:48.520290 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:48.520293 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:48.520512 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:48.520795 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:48.520799 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:48.520801 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:48.520888 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:48.521291 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:44:48.521294 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:44:48.521393 (+ 99us) client_negotiation.cc:769] Sending connection context
0925 04:44:48.521497 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 04:44:48.521654 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":119,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/krb5kdc/oryx_127.0.98.67.keytab.
Entry for principal oryx/127.0.98.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/krb5kdc/oryx_127.0.98.67.keytab.
I20260925 04:44:48.539700 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:0
--local_ip_for_outbound_sockets=127.0.98.67
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:40947
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/krb5kdc/oryx_127.0.98.67.keytab
--principal=oryx/127.0.98.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:37893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:44:48.654310 8270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:44:48.654613 8270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:44:48.654686 8270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:44:48.658869 8270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:44:48.658964 8270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:44:48.659037 8270 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 04:44:48.659123 8270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260925 04:44:48.664426 8270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:37893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/krb5kdc/oryx_127.0.98.67.keytab
--principal=oryx/127.0.98.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.126:40947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:44:48.665524 8270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:44:48.666649 8270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:44:48.676405 8275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:48.676378 8276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:48.676373 8278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:48.676900 8270 server_base.cc:1034] running on GCE node
I20260925 04:44:48.677276 8270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:44:48.677927 8270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:44:48.679267 8270 hybrid_clock.cc:648] HybridClock initialized: now 1790311488679223 us; error 67 us; skew 500 ppm
Sep 25 04:44:48 dist-test-slave-kghz krb5kdc[7873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311488, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:44:48.682240 8270 init.cc:377] Logged in from keytab as oryx/127.0.98.67@KRBTEST.COM (short username oryx)
I20260925 04:44:48.683210 8270 webserver.cc:460] Webserver started at http://127.0.98.67:32951/ using document root <none> and password file <none>
I20260925 04:44:48.683784 8270 fs_manager.cc:359] Metadata directory not provided
I20260925 04:44:48.683874 8270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:44:48.684106 8270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:44:48.686223 8270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/minicluster-data/ts-2/data/instance:
uuid: "7d5cb59e32a844c988bd1754fa8f4550"
format_stamp: "Formatted at 2026-09-25 04:44:48 on dist-test-slave-kghz"
server_key: "44c7f9bdd189a2ce128e1bbc4ab7ac53"
server_key_iv: "6c767fa73a4c2cccbcefc559dfff05ec"
server_key_version: "encryptionkey@0"
I20260925 04:44:48.686864 8270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/minicluster-data/ts-2/wal/instance:
uuid: "7d5cb59e32a844c988bd1754fa8f4550"
format_stamp: "Formatted at 2026-09-25 04:44:48 on dist-test-slave-kghz"
server_key: "44c7f9bdd189a2ce128e1bbc4ab7ac53"
server_key_iv: "6c767fa73a4c2cccbcefc559dfff05ec"
server_key_version: "encryptionkey@0"
I20260925 04:44:48.690721 8270 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 04:44:48.693307 8285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:48.694728 8270 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:44:48.694864 8270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/minicluster-data/ts-2/wal
uuid: "7d5cb59e32a844c988bd1754fa8f4550"
format_stamp: "Formatted at 2026-09-25 04:44:48 on dist-test-slave-kghz"
server_key: "44c7f9bdd189a2ce128e1bbc4ab7ac53"
server_key_iv: "6c767fa73a4c2cccbcefc559dfff05ec"
server_key_version: "encryptionkey@0"
I20260925 04:44:48.694975 8270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:44:48.713208 8270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:44:48.716758 8270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:44:48.716975 8270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:44:48.718495 8270 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:44:48.718564 8270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:48.718631 8270 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:44:48.718670 8270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 04:44:48 dist-test-slave-kghz krb5kdc[7873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311488, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM
I20260925 04:44:48.729475 8270 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:41117
I20260925 04:44:48.729862 8400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:41117 every 8 connection(s)
I20260925 04:44:48.730626 8270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
I20260925 04:44:48.732822 7967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:48.719360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:48777 (local address 127.0.98.126:40947)
0925 04:44:48.719479 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:48.719482 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:48.720563 (+ 1081us) server_negotiation.cc:407] Connection header received
0925 04:44:48.721617 (+ 1054us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:48.721625 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:48.721695 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:48.721796 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:48.723318 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:48.723886 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:48.725753 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:48.726140 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:48.726282 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:48.728872 (+ 2590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:48.728875 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:48.728877 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:48.728902 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:48.730687 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:48.731352 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:48.731358 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:48.731360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:48.731458 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:48.731783 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:48.731785 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:48.731787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:48.731980 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:48.732064 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:48.732595 (+ 531us) server_negotiation.cc:299] Negotiation successful
0925 04:44:48.732704 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 04:44:48.733655 8294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:48.719953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:40947 (local address 127.0.98.67:48777)
0925 04:44:48.720420 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:48.720461 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:48.721341 (+ 880us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:48.722035 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:48.722046 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:48.722599 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:48.723176 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:48.723191 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:48.724022 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:48.724026 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:48.725617 (+ 1591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:48.725631 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:48.726326 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:48.726330 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:48.726503 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:48.727107 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:48.727124 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:48.728695 (+ 1571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:44:48.730828 (+ 2133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:48.730834 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:48.730849 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:48.731219 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:48.731559 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:48.731561 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:48.731563 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:48.731694 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:48.732072 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:44:48.732077 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:44:48.732343 (+ 266us) client_negotiation.cc:769] Sending connection context
0925 04:44:48.732507 (+ 164us) client_negotiation.cc:240] Negotiation successful
0925 04:44:48.732769 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":136,"threads_started":1}
I20260925 04:44:48.734957 8401 heartbeater.cc:347] Connected to a master server at 127.0.98.126:40947
I20260925 04:44:48.735123 8401 heartbeater.cc:464] Registering TS with master...
I20260925 04:44:48.735638 8401 heartbeater.cc:511] Master 127.0.98.126:40947 requested a full tablet report, sending...
I20260925 04:44:48.736747 7917 ts_manager.cc:194] Registered new tserver with Master: 7d5cb59e32a844c988bd1754fa8f4550 (127.0.98.67:41117)
I20260925 04:44:48.736927 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 8270
I20260925 04:44:48.737067 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/minicluster-data/ts-2/wal/instance
I20260925 04:44:48.737393 393 external_mini_cluster.cc:1428] Setting key 6eedd397fba388e438a43196609d8679
I20260925 04:44:48.737543 7917 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.67@KRBTEST.COM'} at 127.0.98.67:48777
I20260925 04:44:48.739540 393 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 04:44:48.745781 8407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:48.737370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:40947 (local address 127.0.98.67:33989)
0925 04:44:48.737643 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:48.737659 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:48.737753 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:48.738002 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:48.738005 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:48.738286 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:48.738418 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:48.738426 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:48.739432 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:48.739435 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:48.740357 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:48.740371 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:48.740937 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:48.740941 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:48.741051 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:48.741643 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:48.741664 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:48.742215 (+ 551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:44:48.744011 (+ 1796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:48.744017 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:48.744020 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:48.744338 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:48.744644 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:48.744649 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:48.744653 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:48.744756 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:48.745323 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:44:48.745325 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:44:48.745413 (+ 88us) client_negotiation.cc:769] Sending connection context
0925 04:44:48.745503 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 04:44:48.745634 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":108,"threads_started":1}
I20260925 04:44:48.745995 7967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:48.737358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:33989 (local address 127.0.98.126:40947)
0925 04:44:48.737780 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:48.737786 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:48.737803 (+ 17us) server_negotiation.cc:407] Connection header received
0925 04:44:48.737841 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:48.737844 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:48.737886 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:48.737963 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:48.738586 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:48.739308 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:48.740520 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:48.740813 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:48.740934 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:48.742390 (+ 1456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:48.742393 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:48.742396 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:48.742418 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:48.743887 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:48.744460 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:48.744463 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:48.744464 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:48.744546 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:48.744848 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:48.744851 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:48.744852 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:48.745208 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:48.745333 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:48.745592 (+ 259us) server_negotiation.cc:299] Negotiation successful
0925 04:44:48.745739 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349}
Sep 25 04:44:48 dist-test-slave-kghz krb5kdc[7873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311488, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 04:44:48 dist-test-slave-kghz krb5kdc[7873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:44:48 dist-test-slave-kghz krb5kdc[7873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
W20260925 04:44:48.765130 7967 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:44:48.758877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35350 (local address 127.0.98.126:40947)
0925 04:44:48.759009 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:48.759012 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:48.759024 (+ 12us) server_negotiation.cc:407] Connection header received
0925 04:44:48.759066 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:48.759068 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:48.759106 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:48.759197 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:48.759911 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:48.760425 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:48.761297 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:48.761512 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:48.761626 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:48.764969 (+ 3343us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35350: BlockingRecv error: recv got EOF from 127.0.0.1:35350 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 25 04:44:48 dist-test-slave-kghz krb5kdc[7873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311488, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM
I20260925 04:44:48.775475 7967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:48.766684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35364 (local address 127.0.98.126:40947)
0925 04:44:48.766864 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:48.766868 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:48.767004 (+ 136us) server_negotiation.cc:407] Connection header received
0925 04:44:48.767198 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:48.767201 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:48.767250 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:48.767343 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:48.767954 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:48.768471 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:48.769279 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:48.769494 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:48.769610 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:48.772256 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:48.772260 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:48.772262 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:48.772289 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:48.773760 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:48.774210 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:48.774215 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:48.774217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:48.774355 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:48.774630 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:48.774634 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:48.774636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:48.774821 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:48.774954 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:48.775183 (+ 229us) server_negotiation.cc:299] Negotiation successful
0925 04:44:48.775298 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260925 04:44:48.778731 7917 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35364:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:44:48.781873 7917 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:44:48.795903 8423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:48.790238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:39747 (local address 127.0.0.1:34876)
0925 04:44:48.790753 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:48.790804 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:48.791417 (+ 613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:48.792124 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:48.792127 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:48.792154 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:44:48.792279 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:48.792286 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:48.793269 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:48.793272 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:48.794440 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:48.794449 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:48.795344 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:48.795347 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:48.795463 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:48.795502 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 04:44:48.795615 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 04:44:48.795751 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"mutex_wait_us":2,"spinlock_wait_cycles":2304,"thread_start_us":73,"threads_started":1}
I20260925 04:44:48.795913 8425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:48.790439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34876 (local address 127.0.98.66:39747)
0925 04:44:48.791381 (+ 942us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:48.791398 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:48.791432 (+ 34us) server_negotiation.cc:407] Connection header received
0925 04:44:48.791588 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:48.791595 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:48.791820 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:48.791972 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:44:48.792407 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:48.793150 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:48.794571 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:48.795222 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:48.795349 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:48.795527 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:48.795669 (+ 142us) server_negotiation.cc:299] Negotiation successful
0925 04:44:48.795767 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":841,"thread_start_us":107,"threads_started":1}
I20260925 04:44:48.796382 8426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:48.790973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:33301 (local address 127.0.0.1:41128)
0925 04:44:48.791691 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:48.791704 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:48.791779 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:48.792419 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:48.792421 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:48.792446 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:44:48.792527 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:48.792537 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:48.793545 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:48.793549 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:48.794762 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:48.794771 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:48.795989 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:48.795992 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:48.796090 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:48.796112 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 04:44:48.796182 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 04:44:48.796250 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"thread_start_us":106,"threads_started":1}
I20260925 04:44:48.796816 8428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:48.791362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41128 (local address 127.0.98.65:33301)
0925 04:44:48.792000 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:48.792006 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:48.792027 (+ 21us) server_negotiation.cc:407] Connection header received
0925 04:44:48.792097 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:48.792104 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:48.792264 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:48.792410 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:44:48.792634 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:48.793422 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:48.794894 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:48.795691 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:48.795796 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:48.795932 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:48.796558 (+ 626us) server_negotiation.cc:299] Negotiation successful
0925 04:44:48.796676 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":76,"threads_started":1}
I20260925 04:44:48.797143 8424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:48.790430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:41117 (local address 127.0.0.1:36804)
0925 04:44:48.791013 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:48.791028 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:48.791277 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:48.792095 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:48.792102 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:48.792123 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:44:48.792292 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:48.792299 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:48.794537 (+ 2238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:48.794541 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:48.795672 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:48.795684 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:48.796757 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:48.796760 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:48.796849 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:48.796878 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 04:44:48.796948 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 04:44:48.797019 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"mutex_wait_us":25,"spinlock_wait_cycles":247168,"thread_start_us":113,"threads_started":1}
I20260925 04:44:48.797986 8427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:48.790956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36804 (local address 127.0.98.67:41117)
0925 04:44:48.791640 (+ 684us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:48.791653 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:48.791673 (+ 20us) server_negotiation.cc:407] Connection header received
0925 04:44:48.791747 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:48.791752 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:48.791949 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:48.792106 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:44:48.793273 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:48.794397 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:48.795820 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:48.796618 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:48.797630 (+ 1012us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:48.797770 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:48.797810 (+ 40us) server_negotiation.cc:299] Negotiation successful
0925 04:44:48.797878 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":150,"threads_started":1}
I20260925 04:44:48.799543 8195 tablet_service.cc:1467] Processing CreateTablet for tablet a29c90ac66fa41c89d26dfc1af6e9234 (DEFAULT_TABLE table=test-table [id=8c323cfba104462f9751ca2b0082d735]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:44:48.799541 8055 tablet_service.cc:1467] Processing CreateTablet for tablet a29c90ac66fa41c89d26dfc1af6e9234 (DEFAULT_TABLE table=test-table [id=8c323cfba104462f9751ca2b0082d735]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:44:48.800710 8055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a29c90ac66fa41c89d26dfc1af6e9234. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:44:48.800710 8195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a29c90ac66fa41c89d26dfc1af6e9234. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:44:48.800436 8335 tablet_service.cc:1467] Processing CreateTablet for tablet a29c90ac66fa41c89d26dfc1af6e9234 (DEFAULT_TABLE table=test-table [id=8c323cfba104462f9751ca2b0082d735]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:44:48.801295 8335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a29c90ac66fa41c89d26dfc1af6e9234. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:44:48.807967 8429 tablet_bootstrap.cc:492] T a29c90ac66fa41c89d26dfc1af6e9234 P 7d5cb59e32a844c988bd1754fa8f4550: Bootstrap starting.
I20260925 04:44:48.809131 8430 tablet_bootstrap.cc:492] T a29c90ac66fa41c89d26dfc1af6e9234 P 7532f3a035704530bff6b9605888f7c7: Bootstrap starting.
I20260925 04:44:48.810128 8429 tablet_bootstrap.cc:654] T a29c90ac66fa41c89d26dfc1af6e9234 P 7d5cb59e32a844c988bd1754fa8f4550: Neither blocks nor log segments found. Creating new log.
I20260925 04:44:48.810552 8431 tablet_bootstrap.cc:492] T a29c90ac66fa41c89d26dfc1af6e9234 P b4886a90d7904435ada30398c846f69e: Bootstrap starting.
I20260925 04:44:48.811002 8429 log.cc:824] T a29c90ac66fa41c89d26dfc1af6e9234 P 7d5cb59e32a844c988bd1754fa8f4550: Log is configured to *not* fsync() on all Append() calls
I20260925 04:44:48.811048 8430 tablet_bootstrap.cc:654] T a29c90ac66fa41c89d26dfc1af6e9234 P 7532f3a035704530bff6b9605888f7c7: Neither blocks nor log segments found. Creating new log.
I20260925 04:44:48.811909 8430 log.cc:824] T a29c90ac66fa41c89d26dfc1af6e9234 P 7532f3a035704530bff6b9605888f7c7: Log is configured to *not* fsync() on all Append() calls
I20260925 04:44:48.812546 8431 tablet_bootstrap.cc:654] T a29c90ac66fa41c89d26dfc1af6e9234 P b4886a90d7904435ada30398c846f69e: Neither blocks nor log segments found. Creating new log.
I20260925 04:44:48.813265 8429 tablet_bootstrap.cc:492] T a29c90ac66fa41c89d26dfc1af6e9234 P 7d5cb59e32a844c988bd1754fa8f4550: No bootstrap required, opened a new log
I20260925 04:44:48.813362 8431 log.cc:824] T a29c90ac66fa41c89d26dfc1af6e9234 P b4886a90d7904435ada30398c846f69e: Log is configured to *not* fsync() on all Append() calls
I20260925 04:44:48.813519 8429 ts_tablet_manager.cc:1397] T a29c90ac66fa41c89d26dfc1af6e9234 P 7d5cb59e32a844c988bd1754fa8f4550: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 04:44:48.813876 8430 tablet_bootstrap.cc:492] T a29c90ac66fa41c89d26dfc1af6e9234 P 7532f3a035704530bff6b9605888f7c7: No bootstrap required, opened a new log
I20260925 04:44:48.814082 8430 ts_tablet_manager.cc:1397] T a29c90ac66fa41c89d26dfc1af6e9234 P 7532f3a035704530bff6b9605888f7c7: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 04:44:48.815282 8431 tablet_bootstrap.cc:492] T a29c90ac66fa41c89d26dfc1af6e9234 P b4886a90d7904435ada30398c846f69e: No bootstrap required, opened a new log
I20260925 04:44:48.815475 8431 ts_tablet_manager.cc:1397] T a29c90ac66fa41c89d26dfc1af6e9234 P b4886a90d7904435ada30398c846f69e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 04:44:48.816892 8429 raft_consensus.cc:348] T a29c90ac66fa41c89d26dfc1af6e9234 P 7d5cb59e32a844c988bd1754fa8f4550 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4886a90d7904435ada30398c846f69e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39747 } } peers { permanent_uuid: "7d5cb59e32a844c988bd1754fa8f4550" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 41117 } } peers { permanent_uuid: "7532f3a035704530bff6b9605888f7c7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33301 } }
I20260925 04:44:48.817229 8429 raft_consensus.cc:374] T a29c90ac66fa41c89d26dfc1af6e9234 P 7d5cb59e32a844c988bd1754fa8f4550 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:44:48.817320 8429 raft_consensus.cc:729] T a29c90ac66fa41c89d26dfc1af6e9234 P 7d5cb59e32a844c988bd1754fa8f4550 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d5cb59e32a844c988bd1754fa8f4550, State: Initialized, Role: FOLLOWER
I20260925 04:44:48.817818 8429 consensus_queue.cc:260] T a29c90ac66fa41c89d26dfc1af6e9234 P 7d5cb59e32a844c988bd1754fa8f4550 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4886a90d7904435ada30398c846f69e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39747 } } peers { permanent_uuid: "7d5cb59e32a844c988bd1754fa8f4550" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 41117 } } peers { permanent_uuid: "7532f3a035704530bff6b9605888f7c7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33301 } }
I20260925 04:44:48.818629 8401 heartbeater.cc:503] Master 127.0.98.126:40947 was elected leader, sending a full tablet report...
I20260925 04:44:48.818903 8429 ts_tablet_manager.cc:1428] T a29c90ac66fa41c89d26dfc1af6e9234 P 7d5cb59e32a844c988bd1754fa8f4550: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260925 04:44:48.819303 8430 raft_consensus.cc:348] T a29c90ac66fa41c89d26dfc1af6e9234 P 7532f3a035704530bff6b9605888f7c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4886a90d7904435ada30398c846f69e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39747 } } peers { permanent_uuid: "7d5cb59e32a844c988bd1754fa8f4550" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 41117 } } peers { permanent_uuid: "7532f3a035704530bff6b9605888f7c7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33301 } }
I20260925 04:44:48.819304 8431 raft_consensus.cc:348] T a29c90ac66fa41c89d26dfc1af6e9234 P b4886a90d7904435ada30398c846f69e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4886a90d7904435ada30398c846f69e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39747 } } peers { permanent_uuid: "7d5cb59e32a844c988bd1754fa8f4550" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 41117 } } peers { permanent_uuid: "7532f3a035704530bff6b9605888f7c7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33301 } }
I20260925 04:44:48.819563 8431 raft_consensus.cc:374] T a29c90ac66fa41c89d26dfc1af6e9234 P b4886a90d7904435ada30398c846f69e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:44:48.819563 8430 raft_consensus.cc:374] T a29c90ac66fa41c89d26dfc1af6e9234 P 7532f3a035704530bff6b9605888f7c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:44:48.819630 8430 raft_consensus.cc:729] T a29c90ac66fa41c89d26dfc1af6e9234 P 7532f3a035704530bff6b9605888f7c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7532f3a035704530bff6b9605888f7c7, State: Initialized, Role: FOLLOWER
I20260925 04:44:48.819651 8431 raft_consensus.cc:729] T a29c90ac66fa41c89d26dfc1af6e9234 P b4886a90d7904435ada30398c846f69e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4886a90d7904435ada30398c846f69e, State: Initialized, Role: FOLLOWER
I20260925 04:44:48.820063 8430 consensus_queue.cc:260] T a29c90ac66fa41c89d26dfc1af6e9234 P 7532f3a035704530bff6b9605888f7c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4886a90d7904435ada30398c846f69e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39747 } } peers { permanent_uuid: "7d5cb59e32a844c988bd1754fa8f4550" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 41117 } } peers { permanent_uuid: "7532f3a035704530bff6b9605888f7c7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33301 } }
I20260925 04:44:48.820063 8431 consensus_queue.cc:260] T a29c90ac66fa41c89d26dfc1af6e9234 P b4886a90d7904435ada30398c846f69e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4886a90d7904435ada30398c846f69e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39747 } } peers { permanent_uuid: "7d5cb59e32a844c988bd1754fa8f4550" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 41117 } } peers { permanent_uuid: "7532f3a035704530bff6b9605888f7c7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33301 } }
I20260925 04:44:48.820670 8121 heartbeater.cc:503] Master 127.0.98.126:40947 was elected leader, sending a full tablet report...
I20260925 04:44:48.821004 8430 ts_tablet_manager.cc:1428] T a29c90ac66fa41c89d26dfc1af6e9234 P 7532f3a035704530bff6b9605888f7c7: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260925 04:44:48.821030 8431 ts_tablet_manager.cc:1428] T a29c90ac66fa41c89d26dfc1af6e9234 P b4886a90d7904435ada30398c846f69e: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260925 04:44:48.821429 8261 heartbeater.cc:503] Master 127.0.98.126:40947 was elected leader, sending a full tablet report...
I20260925 04:44:48.833089 8435 raft_consensus.cc:482] T a29c90ac66fa41c89d26dfc1af6e9234 P 7d5cb59e32a844c988bd1754fa8f4550 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:44:48.833324 8435 raft_consensus.cc:504] T a29c90ac66fa41c89d26dfc1af6e9234 P 7d5cb59e32a844c988bd1754fa8f4550 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4886a90d7904435ada30398c846f69e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39747 } } peers { permanent_uuid: "7d5cb59e32a844c988bd1754fa8f4550" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 41117 } } peers { permanent_uuid: "7532f3a035704530bff6b9605888f7c7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33301 } }
I20260925 04:44:48.834493 8435 leader_election.cc:288] T a29c90ac66fa41c89d26dfc1af6e9234 P 7d5cb59e32a844c988bd1754fa8f4550 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b4886a90d7904435ada30398c846f69e (127.0.98.66:39747), 7532f3a035704530bff6b9605888f7c7 (127.0.98.65:33301)
I20260925 04:44:48.838446 8294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:48.834750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:33301 (local address 127.0.98.67:44207)
0925 04:44:48.834972 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:48.834989 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:48.835138 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:48.835498 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:48.835501 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:48.835516 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:44:48.835630 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:48.835636 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:48.836504 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:48.836511 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:48.837303 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:48.837311 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:48.837974 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:48.837977 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:48.838081 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:48.838103 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 04:44:48.838191 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 04:44:48.838286 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121}
I20260925 04:44:48.838426 8428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:48.834848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:44207 (local address 127.0.98.65:33301)
0925 04:44:48.835007 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:48.835014 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:48.835226 (+ 212us) server_negotiation.cc:407] Connection header received
0925 04:44:48.835328 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:48.835332 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:48.835377 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:48.835458 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:44:48.835736 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:48.836409 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:48.837427 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:48.837853 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:48.837978 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:48.838043 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:48.838207 (+ 164us) server_negotiation.cc:299] Negotiation successful
0925 04:44:48.838305 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260925 04:44:48.838673 8425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:48.834863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:58081 (local address 127.0.98.66:39747)
0925 04:44:48.834984 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:48.834989 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:48.835299 (+ 310us) server_negotiation.cc:407] Connection header received
0925 04:44:48.835452 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:48.835454 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:48.835504 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:48.835601 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:44:48.836237 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:48.836737 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:48.837739 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:48.838127 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:48.838240 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:48.838319 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:48.838482 (+ 163us) server_negotiation.cc:299] Negotiation successful
0925 04:44:48.838560 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 04:44:48.838689 8438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:48.834750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:39747 (local address 127.0.98.67:58081)
0925 04:44:48.835206 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:48.835217 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:48.835352 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:48.835608 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:48.835611 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:48.835627 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:44:48.836107 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:48.836118 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:48.836850 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:48.836856 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:48.837612 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:48.837620 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:48.838264 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:48.838269 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:48.838367 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:48.838383 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 04:44:48.838467 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 04:44:48.838534 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"mutex_wait_us":43,"thread_start_us":143,"threads_started":1}
I20260925 04:44:48.839197 8075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a29c90ac66fa41c89d26dfc1af6e9234" candidate_uuid: "7d5cb59e32a844c988bd1754fa8f4550" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7532f3a035704530bff6b9605888f7c7" is_pre_election: true
I20260925 04:44:48.839216 8215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a29c90ac66fa41c89d26dfc1af6e9234" candidate_uuid: "7d5cb59e32a844c988bd1754fa8f4550" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4886a90d7904435ada30398c846f69e" is_pre_election: true
I20260925 04:44:48.839494 8215 raft_consensus.cc:2454] T a29c90ac66fa41c89d26dfc1af6e9234 P b4886a90d7904435ada30398c846f69e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d5cb59e32a844c988bd1754fa8f4550 in term 0.
I20260925 04:44:48.839498 8075 raft_consensus.cc:2454] T a29c90ac66fa41c89d26dfc1af6e9234 P 7532f3a035704530bff6b9605888f7c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d5cb59e32a844c988bd1754fa8f4550 in term 0.
I20260925 04:44:48.839980 8286 leader_election.cc:302] T a29c90ac66fa41c89d26dfc1af6e9234 P 7d5cb59e32a844c988bd1754fa8f4550 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d5cb59e32a844c988bd1754fa8f4550, b4886a90d7904435ada30398c846f69e; no voters:
I20260925 04:44:48.840252 8435 raft_consensus.cc:2781] T a29c90ac66fa41c89d26dfc1af6e9234 P 7d5cb59e32a844c988bd1754fa8f4550 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:44:48.840328 8435 raft_consensus.cc:482] T a29c90ac66fa41c89d26dfc1af6e9234 P 7d5cb59e32a844c988bd1754fa8f4550 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:44:48.840376 8435 raft_consensus.cc:3037] T a29c90ac66fa41c89d26dfc1af6e9234 P 7d5cb59e32a844c988bd1754fa8f4550 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:44:48.841516 8435 raft_consensus.cc:504] T a29c90ac66fa41c89d26dfc1af6e9234 P 7d5cb59e32a844c988bd1754fa8f4550 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4886a90d7904435ada30398c846f69e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39747 } } peers { permanent_uuid: "7d5cb59e32a844c988bd1754fa8f4550" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 41117 } } peers { permanent_uuid: "7532f3a035704530bff6b9605888f7c7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33301 } }
I20260925 04:44:48.841919 8435 leader_election.cc:288] T a29c90ac66fa41c89d26dfc1af6e9234 P 7d5cb59e32a844c988bd1754fa8f4550 [CANDIDATE]: Term 1 election: Requested vote from peers b4886a90d7904435ada30398c846f69e (127.0.98.66:39747), 7532f3a035704530bff6b9605888f7c7 (127.0.98.65:33301)
I20260925 04:44:48.842345 8215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a29c90ac66fa41c89d26dfc1af6e9234" candidate_uuid: "7d5cb59e32a844c988bd1754fa8f4550" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4886a90d7904435ada30398c846f69e"
I20260925 04:44:48.842388 8075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a29c90ac66fa41c89d26dfc1af6e9234" candidate_uuid: "7d5cb59e32a844c988bd1754fa8f4550" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7532f3a035704530bff6b9605888f7c7"
I20260925 04:44:48.842495 8075 raft_consensus.cc:3037] T a29c90ac66fa41c89d26dfc1af6e9234 P 7532f3a035704530bff6b9605888f7c7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:44:48.842489 8215 raft_consensus.cc:3037] T a29c90ac66fa41c89d26dfc1af6e9234 P b4886a90d7904435ada30398c846f69e [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:44:48.843822 8215 raft_consensus.cc:2454] T a29c90ac66fa41c89d26dfc1af6e9234 P b4886a90d7904435ada30398c846f69e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d5cb59e32a844c988bd1754fa8f4550 in term 1.
I20260925 04:44:48.843822 8075 raft_consensus.cc:2454] T a29c90ac66fa41c89d26dfc1af6e9234 P 7532f3a035704530bff6b9605888f7c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d5cb59e32a844c988bd1754fa8f4550 in term 1.
I20260925 04:44:48.844127 8286 leader_election.cc:302] T a29c90ac66fa41c89d26dfc1af6e9234 P 7d5cb59e32a844c988bd1754fa8f4550 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 7532f3a035704530bff6b9605888f7c7, 7d5cb59e32a844c988bd1754fa8f4550, b4886a90d7904435ada30398c846f69e; no voters:
I20260925 04:44:48.844277 8435 raft_consensus.cc:2781] T a29c90ac66fa41c89d26dfc1af6e9234 P 7d5cb59e32a844c988bd1754fa8f4550 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:44:48.844484 8435 raft_consensus.cc:686] T a29c90ac66fa41c89d26dfc1af6e9234 P 7d5cb59e32a844c988bd1754fa8f4550 [term 1 LEADER]: Becoming Leader. State: Replica: 7d5cb59e32a844c988bd1754fa8f4550, State: Running, Role: LEADER
I20260925 04:44:48.844816 8435 consensus_queue.cc:237] T a29c90ac66fa41c89d26dfc1af6e9234 P 7d5cb59e32a844c988bd1754fa8f4550 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4886a90d7904435ada30398c846f69e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39747 } } peers { permanent_uuid: "7d5cb59e32a844c988bd1754fa8f4550" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 41117 } } peers { permanent_uuid: "7532f3a035704530bff6b9605888f7c7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33301 } }
I20260925 04:44:48.848292 7917 catalog_manager.cc:5495] T a29c90ac66fa41c89d26dfc1af6e9234 P 7d5cb59e32a844c988bd1754fa8f4550 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7d5cb59e32a844c988bd1754fa8f4550 (127.0.98.67). New cstate: current_term: 1 leader_uuid: "7d5cb59e32a844c988bd1754fa8f4550" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4886a90d7904435ada30398c846f69e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d5cb59e32a844c988bd1754fa8f4550" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 41117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7532f3a035704530bff6b9605888f7c7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33301 } health_report { overall_health: UNKNOWN } } }
I20260925 04:44:48.864727 8427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:48.861186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36820 (local address 127.0.98.67:41117)
0925 04:44:48.861320 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:48.861326 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:48.861350 (+ 24us) server_negotiation.cc:407] Connection header received
0925 04:44:48.861434 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:48.861437 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:48.861504 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:48.861613 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:44:48.861853 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:48.862414 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:48.863280 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:48.863506 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:48.863609 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:48.863869 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:44:48.864309 (+ 440us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:44:48.864429 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:48.864493 (+ 64us) server_negotiation.cc:299] Negotiation successful
0925 04:44:48.864547 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 04:44:48.870810 8075 raft_consensus.cc:1261] T a29c90ac66fa41c89d26dfc1af6e9234 P 7532f3a035704530bff6b9605888f7c7 [term 1 FOLLOWER]: Refusing update from remote peer 7d5cb59e32a844c988bd1754fa8f4550: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:44:48.870915 8215 raft_consensus.cc:1261] T a29c90ac66fa41c89d26dfc1af6e9234 P b4886a90d7904435ada30398c846f69e [term 1 FOLLOWER]: Refusing update from remote peer 7d5cb59e32a844c988bd1754fa8f4550: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:44:48.871640 8435 consensus_queue.cc:1035] T a29c90ac66fa41c89d26dfc1af6e9234 P 7d5cb59e32a844c988bd1754fa8f4550 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7532f3a035704530bff6b9605888f7c7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:44:48.871855 8439 consensus_queue.cc:1035] T a29c90ac66fa41c89d26dfc1af6e9234 P 7d5cb59e32a844c988bd1754fa8f4550 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4886a90d7904435ada30398c846f69e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:44:48.880313 8443 mvcc.cc:205] Tried to move back new op lower bound from 7333115858406850560 to 7333115858311299072. Current Snapshot: MvccSnapshot[applied={T|T < 7333115858406850560}]
I20260925 04:44:48.881120 8444 mvcc.cc:205] Tried to move back new op lower bound from 7333115858406850560 to 7333115858311299072. Current Snapshot: MvccSnapshot[applied={T|T < 7333115858406850560}]
I20260925 04:44:48.889267 7917 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35364:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:44:48.889426 7917 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35364:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:44:48.891885 7917 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1f45ce55bb3f415c8d7178965484a569: Sending DeleteTablet for 3 replicas of tablet a29c90ac66fa41c89d26dfc1af6e9234
I20260925 04:44:48.892719 8335 tablet_service.cc:1514] Processing DeleteTablet for tablet a29c90ac66fa41c89d26dfc1af6e9234 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:44:48 UTC) from {username='oryx', principal='oryx/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:36804
I20260925 04:44:48.892736 8195 tablet_service.cc:1514] Processing DeleteTablet for tablet a29c90ac66fa41c89d26dfc1af6e9234 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:44:48 UTC) from {username='oryx', principal='oryx/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:34876
I20260925 04:44:48.892834 8055 tablet_service.cc:1514] Processing DeleteTablet for tablet a29c90ac66fa41c89d26dfc1af6e9234 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:44:48 UTC) from {username='oryx', principal='oryx/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:41128
I20260925 04:44:48.893218 8452 tablet_replica.cc:330] T a29c90ac66fa41c89d26dfc1af6e9234 P 7d5cb59e32a844c988bd1754fa8f4550: stopping tablet replica
I20260925 04:44:48.893535 8452 raft_consensus.cc:2229] T a29c90ac66fa41c89d26dfc1af6e9234 P 7d5cb59e32a844c988bd1754fa8f4550 [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:44:48.893623 7917 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35364:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 04:44:48.893945 8452 raft_consensus.cc:2258] T a29c90ac66fa41c89d26dfc1af6e9234 P 7d5cb59e32a844c988bd1754fa8f4550 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260925 04:44:48.894028 7917 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:44:48.895417 8452 ts_tablet_manager.cc:1905] T a29c90ac66fa41c89d26dfc1af6e9234 P 7d5cb59e32a844c988bd1754fa8f4550: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:44:48.898360 8452 ts_tablet_manager.cc:1918] T a29c90ac66fa41c89d26dfc1af6e9234 P 7d5cb59e32a844c988bd1754fa8f4550: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:44:48.898481 8452 log.cc:1196] T a29c90ac66fa41c89d26dfc1af6e9234 P 7d5cb59e32a844c988bd1754fa8f4550: Deleting WAL directory at /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/minicluster-data/ts-2/wal/wals/a29c90ac66fa41c89d26dfc1af6e9234
I20260925 04:44:48.898849 8452 ts_tablet_manager.cc:1939] T a29c90ac66fa41c89d26dfc1af6e9234 P 7d5cb59e32a844c988bd1754fa8f4550: Deleting consensus metadata
I20260925 04:44:48.899300 8335 tablet_service.cc:1467] Processing CreateTablet for tablet 1b25a2924d674b11b8de554694498895 (DEFAULT_TABLE table=test-table [id=f12b58e6b4254609838581aed53b1c02]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:44:48.899348 8195 tablet_service.cc:1467] Processing CreateTablet for tablet 1b25a2924d674b11b8de554694498895 (DEFAULT_TABLE table=test-table [id=f12b58e6b4254609838581aed53b1c02]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:44:48.899602 8335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b25a2924d674b11b8de554694498895. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:44:48.899641 8195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b25a2924d674b11b8de554694498895. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:44:48.901360 8429 tablet_bootstrap.cc:492] T 1b25a2924d674b11b8de554694498895 P 7d5cb59e32a844c988bd1754fa8f4550: Bootstrap starting.
I20260925 04:44:48.902480 8429 tablet_bootstrap.cc:654] T 1b25a2924d674b11b8de554694498895 P 7d5cb59e32a844c988bd1754fa8f4550: Neither blocks nor log segments found. Creating new log.
I20260925 04:44:48.902472 8431 tablet_bootstrap.cc:492] T 1b25a2924d674b11b8de554694498895 P b4886a90d7904435ada30398c846f69e: Bootstrap starting.
I20260925 04:44:48.902581 8055 tablet_service.cc:1467] Processing CreateTablet for tablet 1b25a2924d674b11b8de554694498895 (DEFAULT_TABLE table=test-table [id=f12b58e6b4254609838581aed53b1c02]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:44:48.902659 7903 catalog_manager.cc:4841] TS 7d5cb59e32a844c988bd1754fa8f4550 (127.0.98.67:41117): tablet a29c90ac66fa41c89d26dfc1af6e9234 (table test-table [id=8c323cfba104462f9751ca2b0082d735]) successfully deleted
I20260925 04:44:48.902848 8055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b25a2924d674b11b8de554694498895. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:44:48.903470 8431 tablet_bootstrap.cc:654] T 1b25a2924d674b11b8de554694498895 P b4886a90d7904435ada30398c846f69e: Neither blocks nor log segments found. Creating new log.
I20260925 04:44:48.903692 8429 tablet_bootstrap.cc:492] T 1b25a2924d674b11b8de554694498895 P 7d5cb59e32a844c988bd1754fa8f4550: No bootstrap required, opened a new log
I20260925 04:44:48.903787 8429 ts_tablet_manager.cc:1397] T 1b25a2924d674b11b8de554694498895 P 7d5cb59e32a844c988bd1754fa8f4550: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260925 04:44:48.904422 8429 raft_consensus.cc:348] T 1b25a2924d674b11b8de554694498895 P 7d5cb59e32a844c988bd1754fa8f4550 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7532f3a035704530bff6b9605888f7c7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33301 } } peers { permanent_uuid: "b4886a90d7904435ada30398c846f69e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39747 } } peers { permanent_uuid: "7d5cb59e32a844c988bd1754fa8f4550" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 41117 } }
I20260925 04:44:48.904541 8429 raft_consensus.cc:374] T 1b25a2924d674b11b8de554694498895 P 7d5cb59e32a844c988bd1754fa8f4550 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:44:48.904577 8429 raft_consensus.cc:729] T 1b25a2924d674b11b8de554694498895 P 7d5cb59e32a844c988bd1754fa8f4550 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d5cb59e32a844c988bd1754fa8f4550, State: Initialized, Role: FOLLOWER
I20260925 04:44:48.904702 8431 tablet_bootstrap.cc:492] T 1b25a2924d674b11b8de554694498895 P b4886a90d7904435ada30398c846f69e: No bootstrap required, opened a new log
I20260925 04:44:48.904701 8429 consensus_queue.cc:260] T 1b25a2924d674b11b8de554694498895 P 7d5cb59e32a844c988bd1754fa8f4550 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7532f3a035704530bff6b9605888f7c7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33301 } } peers { permanent_uuid: "b4886a90d7904435ada30398c846f69e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39747 } } peers { permanent_uuid: "7d5cb59e32a844c988bd1754fa8f4550" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 41117 } }
I20260925 04:44:48.904773 8431 ts_tablet_manager.cc:1397] T 1b25a2924d674b11b8de554694498895 P b4886a90d7904435ada30398c846f69e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 04:44:48.904978 8429 ts_tablet_manager.cc:1428] T 1b25a2924d674b11b8de554694498895 P 7d5cb59e32a844c988bd1754fa8f4550: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260925 04:44:48.905225 8431 raft_consensus.cc:348] T 1b25a2924d674b11b8de554694498895 P b4886a90d7904435ada30398c846f69e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7532f3a035704530bff6b9605888f7c7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33301 } } peers { permanent_uuid: "b4886a90d7904435ada30398c846f69e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39747 } } peers { permanent_uuid: "7d5cb59e32a844c988bd1754fa8f4550" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 41117 } }
I20260925 04:44:48.905318 8431 raft_consensus.cc:374] T 1b25a2924d674b11b8de554694498895 P b4886a90d7904435ada30398c846f69e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:44:48.905339 8431 raft_consensus.cc:729] T 1b25a2924d674b11b8de554694498895 P b4886a90d7904435ada30398c846f69e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4886a90d7904435ada30398c846f69e, State: Initialized, Role: FOLLOWER
I20260925 04:44:48.905474 8431 consensus_queue.cc:260] T 1b25a2924d674b11b8de554694498895 P b4886a90d7904435ada30398c846f69e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7532f3a035704530bff6b9605888f7c7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33301 } } peers { permanent_uuid: "b4886a90d7904435ada30398c846f69e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39747 } } peers { permanent_uuid: "7d5cb59e32a844c988bd1754fa8f4550" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 41117 } }
I20260925 04:44:48.905655 8430 tablet_bootstrap.cc:492] T 1b25a2924d674b11b8de554694498895 P 7532f3a035704530bff6b9605888f7c7: Bootstrap starting.
I20260925 04:44:48.905750 8431 ts_tablet_manager.cc:1428] T 1b25a2924d674b11b8de554694498895 P b4886a90d7904435ada30398c846f69e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 04:44:48.906167 8454 tablet_replica.cc:330] T a29c90ac66fa41c89d26dfc1af6e9234 P 7532f3a035704530bff6b9605888f7c7: stopping tablet replica
I20260925 04:44:48.906548 8454 raft_consensus.cc:2229] T a29c90ac66fa41c89d26dfc1af6e9234 P 7532f3a035704530bff6b9605888f7c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:44:48.906647 8430 tablet_bootstrap.cc:654] T 1b25a2924d674b11b8de554694498895 P 7532f3a035704530bff6b9605888f7c7: Neither blocks nor log segments found. Creating new log.
I20260925 04:44:48.906896 8454 raft_consensus.cc:2258] T a29c90ac66fa41c89d26dfc1af6e9234 P 7532f3a035704530bff6b9605888f7c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:44:48.907462 8453 tablet_replica.cc:330] T a29c90ac66fa41c89d26dfc1af6e9234 P b4886a90d7904435ada30398c846f69e: stopping tablet replica
I20260925 04:44:48.907760 8453 raft_consensus.cc:2229] T a29c90ac66fa41c89d26dfc1af6e9234 P b4886a90d7904435ada30398c846f69e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:44:48.908080 8453 raft_consensus.cc:2258] T a29c90ac66fa41c89d26dfc1af6e9234 P b4886a90d7904435ada30398c846f69e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:44:48.909112 8430 tablet_bootstrap.cc:492] T 1b25a2924d674b11b8de554694498895 P 7532f3a035704530bff6b9605888f7c7: No bootstrap required, opened a new log
I20260925 04:44:48.909216 8430 ts_tablet_manager.cc:1397] T 1b25a2924d674b11b8de554694498895 P 7532f3a035704530bff6b9605888f7c7: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260925 04:44:48.909310 8454 ts_tablet_manager.cc:1905] T a29c90ac66fa41c89d26dfc1af6e9234 P 7532f3a035704530bff6b9605888f7c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:44:48.909654 8430 raft_consensus.cc:348] T 1b25a2924d674b11b8de554694498895 P 7532f3a035704530bff6b9605888f7c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7532f3a035704530bff6b9605888f7c7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33301 } } peers { permanent_uuid: "b4886a90d7904435ada30398c846f69e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39747 } } peers { permanent_uuid: "7d5cb59e32a844c988bd1754fa8f4550" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 41117 } }
I20260925 04:44:48.909766 8430 raft_consensus.cc:374] T 1b25a2924d674b11b8de554694498895 P 7532f3a035704530bff6b9605888f7c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:44:48.909809 8430 raft_consensus.cc:729] T 1b25a2924d674b11b8de554694498895 P 7532f3a035704530bff6b9605888f7c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7532f3a035704530bff6b9605888f7c7, State: Initialized, Role: FOLLOWER
I20260925 04:44:48.909934 8430 consensus_queue.cc:260] T 1b25a2924d674b11b8de554694498895 P 7532f3a035704530bff6b9605888f7c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7532f3a035704530bff6b9605888f7c7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33301 } } peers { permanent_uuid: "b4886a90d7904435ada30398c846f69e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39747 } } peers { permanent_uuid: "7d5cb59e32a844c988bd1754fa8f4550" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 41117 } }
I20260925 04:44:48.910216 8430 ts_tablet_manager.cc:1428] T 1b25a2924d674b11b8de554694498895 P 7532f3a035704530bff6b9605888f7c7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 04:44:48.912436 8454 ts_tablet_manager.cc:1918] T a29c90ac66fa41c89d26dfc1af6e9234 P 7532f3a035704530bff6b9605888f7c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:44:48.912560 8454 log.cc:1196] T a29c90ac66fa41c89d26dfc1af6e9234 P 7532f3a035704530bff6b9605888f7c7: Deleting WAL directory at /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/minicluster-data/ts-0/wal/wals/a29c90ac66fa41c89d26dfc1af6e9234
I20260925 04:44:48.912881 8454 ts_tablet_manager.cc:1939] T a29c90ac66fa41c89d26dfc1af6e9234 P 7532f3a035704530bff6b9605888f7c7: Deleting consensus metadata
I20260925 04:44:48.914101 7905 catalog_manager.cc:4841] TS 7532f3a035704530bff6b9605888f7c7 (127.0.98.65:33301): tablet a29c90ac66fa41c89d26dfc1af6e9234 (table test-table [id=8c323cfba104462f9751ca2b0082d735]) successfully deleted
I20260925 04:44:48.915498 8453 ts_tablet_manager.cc:1905] T a29c90ac66fa41c89d26dfc1af6e9234 P b4886a90d7904435ada30398c846f69e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:44:48.918085 8453 ts_tablet_manager.cc:1918] T a29c90ac66fa41c89d26dfc1af6e9234 P b4886a90d7904435ada30398c846f69e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:44:48.918195 8453 log.cc:1196] T a29c90ac66fa41c89d26dfc1af6e9234 P b4886a90d7904435ada30398c846f69e: Deleting WAL directory at /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/minicluster-data/ts-1/wal/wals/a29c90ac66fa41c89d26dfc1af6e9234
I20260925 04:44:48.918529 8453 ts_tablet_manager.cc:1939] T a29c90ac66fa41c89d26dfc1af6e9234 P b4886a90d7904435ada30398c846f69e: Deleting consensus metadata
I20260925 04:44:48.919739 7902 catalog_manager.cc:4841] TS b4886a90d7904435ada30398c846f69e (127.0.98.66:39747): tablet a29c90ac66fa41c89d26dfc1af6e9234 (table test-table [id=8c323cfba104462f9751ca2b0082d735]) successfully deleted
I20260925 04:44:48.975600 8435 raft_consensus.cc:482] T 1b25a2924d674b11b8de554694498895 P 7d5cb59e32a844c988bd1754fa8f4550 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:44:48.975745 8435 raft_consensus.cc:504] T 1b25a2924d674b11b8de554694498895 P 7d5cb59e32a844c988bd1754fa8f4550 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7532f3a035704530bff6b9605888f7c7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33301 } } peers { permanent_uuid: "b4886a90d7904435ada30398c846f69e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39747 } } peers { permanent_uuid: "7d5cb59e32a844c988bd1754fa8f4550" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 41117 } }
I20260925 04:44:48.976224 8435 leader_election.cc:288] T 1b25a2924d674b11b8de554694498895 P 7d5cb59e32a844c988bd1754fa8f4550 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7532f3a035704530bff6b9605888f7c7 (127.0.98.65:33301), b4886a90d7904435ada30398c846f69e (127.0.98.66:39747)
I20260925 04:44:48.976683 8215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b25a2924d674b11b8de554694498895" candidate_uuid: "7d5cb59e32a844c988bd1754fa8f4550" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4886a90d7904435ada30398c846f69e" is_pre_election: true
I20260925 04:44:48.976693 8075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b25a2924d674b11b8de554694498895" candidate_uuid: "7d5cb59e32a844c988bd1754fa8f4550" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7532f3a035704530bff6b9605888f7c7" is_pre_election: true
I20260925 04:44:48.976853 8075 raft_consensus.cc:2454] T 1b25a2924d674b11b8de554694498895 P 7532f3a035704530bff6b9605888f7c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d5cb59e32a844c988bd1754fa8f4550 in term 0.
I20260925 04:44:48.976845 8215 raft_consensus.cc:2454] T 1b25a2924d674b11b8de554694498895 P b4886a90d7904435ada30398c846f69e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d5cb59e32a844c988bd1754fa8f4550 in term 0.
I20260925 04:44:48.977170 8286 leader_election.cc:302] T 1b25a2924d674b11b8de554694498895 P 7d5cb59e32a844c988bd1754fa8f4550 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d5cb59e32a844c988bd1754fa8f4550, b4886a90d7904435ada30398c846f69e; no voters:
I20260925 04:44:48.977358 8435 raft_consensus.cc:2781] T 1b25a2924d674b11b8de554694498895 P 7d5cb59e32a844c988bd1754fa8f4550 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:44:48.977458 8435 raft_consensus.cc:482] T 1b25a2924d674b11b8de554694498895 P 7d5cb59e32a844c988bd1754fa8f4550 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:44:48.977523 8435 raft_consensus.cc:3037] T 1b25a2924d674b11b8de554694498895 P 7d5cb59e32a844c988bd1754fa8f4550 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:44:48.978744 8435 raft_consensus.cc:504] T 1b25a2924d674b11b8de554694498895 P 7d5cb59e32a844c988bd1754fa8f4550 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7532f3a035704530bff6b9605888f7c7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33301 } } peers { permanent_uuid: "b4886a90d7904435ada30398c846f69e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39747 } } peers { permanent_uuid: "7d5cb59e32a844c988bd1754fa8f4550" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 41117 } }
I20260925 04:44:48.979174 8435 leader_election.cc:288] T 1b25a2924d674b11b8de554694498895 P 7d5cb59e32a844c988bd1754fa8f4550 [CANDIDATE]: Term 1 election: Requested vote from peers 7532f3a035704530bff6b9605888f7c7 (127.0.98.65:33301), b4886a90d7904435ada30398c846f69e (127.0.98.66:39747)
I20260925 04:44:48.979552 8075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b25a2924d674b11b8de554694498895" candidate_uuid: "7d5cb59e32a844c988bd1754fa8f4550" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7532f3a035704530bff6b9605888f7c7"
I20260925 04:44:48.979559 8215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b25a2924d674b11b8de554694498895" candidate_uuid: "7d5cb59e32a844c988bd1754fa8f4550" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4886a90d7904435ada30398c846f69e"
I20260925 04:44:48.979698 8075 raft_consensus.cc:3037] T 1b25a2924d674b11b8de554694498895 P 7532f3a035704530bff6b9605888f7c7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:44:48.979704 8215 raft_consensus.cc:3037] T 1b25a2924d674b11b8de554694498895 P b4886a90d7904435ada30398c846f69e [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:44:48.980602 8215 raft_consensus.cc:2454] T 1b25a2924d674b11b8de554694498895 P b4886a90d7904435ada30398c846f69e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d5cb59e32a844c988bd1754fa8f4550 in term 1.
I20260925 04:44:48.980814 8075 raft_consensus.cc:2454] T 1b25a2924d674b11b8de554694498895 P 7532f3a035704530bff6b9605888f7c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d5cb59e32a844c988bd1754fa8f4550 in term 1.
I20260925 04:44:48.980922 8286 leader_election.cc:302] T 1b25a2924d674b11b8de554694498895 P 7d5cb59e32a844c988bd1754fa8f4550 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d5cb59e32a844c988bd1754fa8f4550, b4886a90d7904435ada30398c846f69e; no voters:
I20260925 04:44:48.981103 8435 raft_consensus.cc:2781] T 1b25a2924d674b11b8de554694498895 P 7d5cb59e32a844c988bd1754fa8f4550 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:44:48.981190 8435 raft_consensus.cc:686] T 1b25a2924d674b11b8de554694498895 P 7d5cb59e32a844c988bd1754fa8f4550 [term 1 LEADER]: Becoming Leader. State: Replica: 7d5cb59e32a844c988bd1754fa8f4550, State: Running, Role: LEADER
I20260925 04:44:48.981350 8435 consensus_queue.cc:237] T 1b25a2924d674b11b8de554694498895 P 7d5cb59e32a844c988bd1754fa8f4550 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7532f3a035704530bff6b9605888f7c7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33301 } } peers { permanent_uuid: "b4886a90d7904435ada30398c846f69e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39747 } } peers { permanent_uuid: "7d5cb59e32a844c988bd1754fa8f4550" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 41117 } }
W20260925 04:44:48.981654 8402 tablet.cc:2369] T 1b25a2924d674b11b8de554694498895 P 7d5cb59e32a844c988bd1754fa8f4550: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:44:48.983072 7917 catalog_manager.cc:5495] T 1b25a2924d674b11b8de554694498895 P 7d5cb59e32a844c988bd1754fa8f4550 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7d5cb59e32a844c988bd1754fa8f4550 (127.0.98.67). New cstate: current_term: 1 leader_uuid: "7d5cb59e32a844c988bd1754fa8f4550" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7532f3a035704530bff6b9605888f7c7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4886a90d7904435ada30398c846f69e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d5cb59e32a844c988bd1754fa8f4550" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 41117 } health_report { overall_health: HEALTHY } } }
W20260925 04:44:49.008556 8262 tablet.cc:2369] T 1b25a2924d674b11b8de554694498895 P b4886a90d7904435ada30398c846f69e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 04:44:49.026129 8122 tablet.cc:2369] T 1b25a2924d674b11b8de554694498895 P 7532f3a035704530bff6b9605888f7c7: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 04:44:49 dist-test-slave-kghz krb5kdc[7873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311487, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.126@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM
I20260925 04:44:49.045083 8464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:49.036174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:40947 (local address 127.0.0.1:35370)
0925 04:44:49.036530 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:49.036542 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:49.036686 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:49.036937 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:49.036940 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:49.037359 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:49.037457 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:49.037464 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:49.038210 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:49.038216 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:49.038860 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:49.038868 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:49.039294 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:49.039297 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:49.039399 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:49.039983 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:49.039995 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:49.041595 (+ 1600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:44:49.043432 (+ 1837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:49.043441 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:49.043447 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:49.043676 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:49.044027 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:49.044031 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:49.044033 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:49.044124 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:49.044491 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:44:49.044500 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:44:49.044610 (+ 110us) client_negotiation.cc:769] Sending connection context
0925 04:44:49.044719 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 04:44:49.044878 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"mutex_wait_us":5,"thread_start_us":120,"threads_started":1}
I20260925 04:44:49.045083 7967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:49.036228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35370 (local address 127.0.98.126:40947)
0925 04:44:49.036367 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:49.036372 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:49.036622 (+ 250us) server_negotiation.cc:407] Connection header received
0925 04:44:49.036781 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:49.036784 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:49.036827 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:49.036908 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:49.037575 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:49.038104 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:49.038977 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:49.039185 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:49.039289 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:49.041729 (+ 2440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:49.041733 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:49.041735 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:49.041770 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:49.043300 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:49.043770 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:49.043776 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:49.043778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:49.043859 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:49.044213 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:49.044216 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:49.044217 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:49.044391 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:49.044492 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:49.044776 (+ 284us) server_negotiation.cc:299] Negotiation successful
0925 04:44:49.044938 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":84,"server-negotiator.queue_time_us":58}
I20260925 04:44:49.049392 7917 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:35370:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 04:44:49.050179 7917 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:44:49.055392 8055 tablet_service.cc:1467] Processing CreateTablet for tablet bc0fd8a59f974f7593c5af33488baafd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4fa1adf4145d48cba4c1f2d3bb0a205f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 04:44:49.055534 8335 tablet_service.cc:1467] Processing CreateTablet for tablet bc0fd8a59f974f7593c5af33488baafd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4fa1adf4145d48cba4c1f2d3bb0a205f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 04:44:49.055770 8055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc0fd8a59f974f7593c5af33488baafd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:44:49.055874 8335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc0fd8a59f974f7593c5af33488baafd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:44:49.058213 8430 tablet_bootstrap.cc:492] T bc0fd8a59f974f7593c5af33488baafd P 7532f3a035704530bff6b9605888f7c7: Bootstrap starting.
I20260925 04:44:49.058398 8195 tablet_service.cc:1467] Processing CreateTablet for tablet bc0fd8a59f974f7593c5af33488baafd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4fa1adf4145d48cba4c1f2d3bb0a205f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 04:44:49.058562 8429 tablet_bootstrap.cc:492] T bc0fd8a59f974f7593c5af33488baafd P 7d5cb59e32a844c988bd1754fa8f4550: Bootstrap starting.
I20260925 04:44:49.058768 8195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc0fd8a59f974f7593c5af33488baafd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:44:49.059218 8430 tablet_bootstrap.cc:654] T bc0fd8a59f974f7593c5af33488baafd P 7532f3a035704530bff6b9605888f7c7: Neither blocks nor log segments found. Creating new log.
I20260925 04:44:49.059379 8429 tablet_bootstrap.cc:654] T bc0fd8a59f974f7593c5af33488baafd P 7d5cb59e32a844c988bd1754fa8f4550: Neither blocks nor log segments found. Creating new log.
I20260925 04:44:49.060436 8430 tablet_bootstrap.cc:492] T bc0fd8a59f974f7593c5af33488baafd P 7532f3a035704530bff6b9605888f7c7: No bootstrap required, opened a new log
I20260925 04:44:49.060528 8430 ts_tablet_manager.cc:1397] T bc0fd8a59f974f7593c5af33488baafd P 7532f3a035704530bff6b9605888f7c7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 04:44:49.060873 8429 tablet_bootstrap.cc:492] T bc0fd8a59f974f7593c5af33488baafd P 7d5cb59e32a844c988bd1754fa8f4550: No bootstrap required, opened a new log
I20260925 04:44:49.060963 8429 ts_tablet_manager.cc:1397] T bc0fd8a59f974f7593c5af33488baafd P 7d5cb59e32a844c988bd1754fa8f4550: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 04:44:49.061021 8430 raft_consensus.cc:348] T bc0fd8a59f974f7593c5af33488baafd P 7532f3a035704530bff6b9605888f7c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7532f3a035704530bff6b9605888f7c7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33301 } } peers { permanent_uuid: "b4886a90d7904435ada30398c846f69e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39747 } } peers { permanent_uuid: "7d5cb59e32a844c988bd1754fa8f4550" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 41117 } }
I20260925 04:44:49.061143 8430 raft_consensus.cc:374] T bc0fd8a59f974f7593c5af33488baafd P 7532f3a035704530bff6b9605888f7c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:44:49.061182 8430 raft_consensus.cc:729] T bc0fd8a59f974f7593c5af33488baafd P 7532f3a035704530bff6b9605888f7c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7532f3a035704530bff6b9605888f7c7, State: Initialized, Role: FOLLOWER
I20260925 04:44:49.061311 8430 consensus_queue.cc:260] T bc0fd8a59f974f7593c5af33488baafd P 7532f3a035704530bff6b9605888f7c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7532f3a035704530bff6b9605888f7c7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33301 } } peers { permanent_uuid: "b4886a90d7904435ada30398c846f69e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39747 } } peers { permanent_uuid: "7d5cb59e32a844c988bd1754fa8f4550" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 41117 } }
I20260925 04:44:49.061357 8429 raft_consensus.cc:348] T bc0fd8a59f974f7593c5af33488baafd P 7d5cb59e32a844c988bd1754fa8f4550 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7532f3a035704530bff6b9605888f7c7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33301 } } peers { permanent_uuid: "b4886a90d7904435ada30398c846f69e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39747 } } peers { permanent_uuid: "7d5cb59e32a844c988bd1754fa8f4550" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 41117 } }
I20260925 04:44:49.061472 8429 raft_consensus.cc:374] T bc0fd8a59f974f7593c5af33488baafd P 7d5cb59e32a844c988bd1754fa8f4550 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:44:49.061509 8429 raft_consensus.cc:729] T bc0fd8a59f974f7593c5af33488baafd P 7d5cb59e32a844c988bd1754fa8f4550 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d5cb59e32a844c988bd1754fa8f4550, State: Initialized, Role: FOLLOWER
I20260925 04:44:49.061630 8430 ts_tablet_manager.cc:1428] T bc0fd8a59f974f7593c5af33488baafd P 7532f3a035704530bff6b9605888f7c7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 04:44:49.061661 8429 consensus_queue.cc:260] T bc0fd8a59f974f7593c5af33488baafd P 7d5cb59e32a844c988bd1754fa8f4550 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7532f3a035704530bff6b9605888f7c7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33301 } } peers { permanent_uuid: "b4886a90d7904435ada30398c846f69e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39747 } } peers { permanent_uuid: "7d5cb59e32a844c988bd1754fa8f4550" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 41117 } }
I20260925 04:44:49.061759 8431 tablet_bootstrap.cc:492] T bc0fd8a59f974f7593c5af33488baafd P b4886a90d7904435ada30398c846f69e: Bootstrap starting.
I20260925 04:44:49.061951 8429 ts_tablet_manager.cc:1428] T bc0fd8a59f974f7593c5af33488baafd P 7d5cb59e32a844c988bd1754fa8f4550: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260925 04:44:49.061918 8435 tablet_replica.cc:439] T bc0fd8a59f974f7593c5af33488baafd P 7d5cb59e32a844c988bd1754fa8f4550: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7532f3a035704530bff6b9605888f7c7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33301 } } peers { permanent_uuid: "b4886a90d7904435ada30398c846f69e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39747 } } peers { permanent_uuid: "7d5cb59e32a844c988bd1754fa8f4550" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 41117 } } }
I20260925 04:44:49.062057 8435 tablet_replica.cc:442] T bc0fd8a59f974f7593c5af33488baafd P 7d5cb59e32a844c988bd1754fa8f4550: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 04:44:49.062783 8431 tablet_bootstrap.cc:654] T bc0fd8a59f974f7593c5af33488baafd P b4886a90d7904435ada30398c846f69e: Neither blocks nor log segments found. Creating new log.
I20260925 04:44:49.062711 8436 tablet_replica.cc:439] T bc0fd8a59f974f7593c5af33488baafd P 7532f3a035704530bff6b9605888f7c7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7532f3a035704530bff6b9605888f7c7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33301 } } peers { permanent_uuid: "b4886a90d7904435ada30398c846f69e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39747 } } peers { permanent_uuid: "7d5cb59e32a844c988bd1754fa8f4550" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 41117 } } }
I20260925 04:44:49.062849 8436 tablet_replica.cc:442] T bc0fd8a59f974f7593c5af33488baafd P 7532f3a035704530bff6b9605888f7c7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 04:44:49.064078 8431 tablet_bootstrap.cc:492] T bc0fd8a59f974f7593c5af33488baafd P b4886a90d7904435ada30398c846f69e: No bootstrap required, opened a new log
I20260925 04:44:49.064170 8431 ts_tablet_manager.cc:1397] T bc0fd8a59f974f7593c5af33488baafd P b4886a90d7904435ada30398c846f69e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 04:44:49.064603 8431 raft_consensus.cc:348] T bc0fd8a59f974f7593c5af33488baafd P b4886a90d7904435ada30398c846f69e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7532f3a035704530bff6b9605888f7c7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33301 } } peers { permanent_uuid: "b4886a90d7904435ada30398c846f69e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39747 } } peers { permanent_uuid: "7d5cb59e32a844c988bd1754fa8f4550" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 41117 } }
I20260925 04:44:49.064770 8431 raft_consensus.cc:374] T bc0fd8a59f974f7593c5af33488baafd P b4886a90d7904435ada30398c846f69e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:44:49.064812 8431 raft_consensus.cc:729] T bc0fd8a59f974f7593c5af33488baafd P b4886a90d7904435ada30398c846f69e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4886a90d7904435ada30398c846f69e, State: Initialized, Role: FOLLOWER
I20260925 04:44:49.064947 8431 consensus_queue.cc:260] T bc0fd8a59f974f7593c5af33488baafd P b4886a90d7904435ada30398c846f69e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7532f3a035704530bff6b9605888f7c7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33301 } } peers { permanent_uuid: "b4886a90d7904435ada30398c846f69e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39747 } } peers { permanent_uuid: "7d5cb59e32a844c988bd1754fa8f4550" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 41117 } }
I20260925 04:44:49.065215 8431 ts_tablet_manager.cc:1428] T bc0fd8a59f974f7593c5af33488baafd P b4886a90d7904435ada30398c846f69e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 04:44:49.065160 8437 tablet_replica.cc:439] T bc0fd8a59f974f7593c5af33488baafd P b4886a90d7904435ada30398c846f69e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7532f3a035704530bff6b9605888f7c7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33301 } } peers { permanent_uuid: "b4886a90d7904435ada30398c846f69e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39747 } } peers { permanent_uuid: "7d5cb59e32a844c988bd1754fa8f4550" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 41117 } } }
I20260925 04:44:49.065294 8437 tablet_replica.cc:442] T bc0fd8a59f974f7593c5af33488baafd P b4886a90d7904435ada30398c846f69e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 04:44:49.176492 8435 raft_consensus.cc:482] T bc0fd8a59f974f7593c5af33488baafd P 7d5cb59e32a844c988bd1754fa8f4550 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:44:49.176674 8435 raft_consensus.cc:504] T bc0fd8a59f974f7593c5af33488baafd P 7d5cb59e32a844c988bd1754fa8f4550 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7532f3a035704530bff6b9605888f7c7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33301 } } peers { permanent_uuid: "b4886a90d7904435ada30398c846f69e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39747 } } peers { permanent_uuid: "7d5cb59e32a844c988bd1754fa8f4550" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 41117 } }
I20260925 04:44:49.177146 8435 leader_election.cc:288] T bc0fd8a59f974f7593c5af33488baafd P 7d5cb59e32a844c988bd1754fa8f4550 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7532f3a035704530bff6b9605888f7c7 (127.0.98.65:33301), b4886a90d7904435ada30398c846f69e (127.0.98.66:39747)
I20260925 04:44:49.177662 8075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc0fd8a59f974f7593c5af33488baafd" candidate_uuid: "7d5cb59e32a844c988bd1754fa8f4550" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7532f3a035704530bff6b9605888f7c7" is_pre_election: true
I20260925 04:44:49.177660 8215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc0fd8a59f974f7593c5af33488baafd" candidate_uuid: "7d5cb59e32a844c988bd1754fa8f4550" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4886a90d7904435ada30398c846f69e" is_pre_election: true
I20260925 04:44:49.177824 8075 raft_consensus.cc:2454] T bc0fd8a59f974f7593c5af33488baafd P 7532f3a035704530bff6b9605888f7c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d5cb59e32a844c988bd1754fa8f4550 in term 0.
I20260925 04:44:49.177841 8215 raft_consensus.cc:2454] T bc0fd8a59f974f7593c5af33488baafd P b4886a90d7904435ada30398c846f69e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d5cb59e32a844c988bd1754fa8f4550 in term 0.
I20260925 04:44:49.178194 8286 leader_election.cc:302] T bc0fd8a59f974f7593c5af33488baafd P 7d5cb59e32a844c988bd1754fa8f4550 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 7532f3a035704530bff6b9605888f7c7, 7d5cb59e32a844c988bd1754fa8f4550, b4886a90d7904435ada30398c846f69e; no voters:
I20260925 04:44:49.178423 8435 raft_consensus.cc:2781] T bc0fd8a59f974f7593c5af33488baafd P 7d5cb59e32a844c988bd1754fa8f4550 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:44:49.178509 8435 raft_consensus.cc:482] T bc0fd8a59f974f7593c5af33488baafd P 7d5cb59e32a844c988bd1754fa8f4550 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:44:49.178570 8435 raft_consensus.cc:3037] T bc0fd8a59f974f7593c5af33488baafd P 7d5cb59e32a844c988bd1754fa8f4550 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:44:49.179648 8435 raft_consensus.cc:504] T bc0fd8a59f974f7593c5af33488baafd P 7d5cb59e32a844c988bd1754fa8f4550 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7532f3a035704530bff6b9605888f7c7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33301 } } peers { permanent_uuid: "b4886a90d7904435ada30398c846f69e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39747 } } peers { permanent_uuid: "7d5cb59e32a844c988bd1754fa8f4550" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 41117 } }
I20260925 04:44:49.180027 8435 leader_election.cc:288] T bc0fd8a59f974f7593c5af33488baafd P 7d5cb59e32a844c988bd1754fa8f4550 [CANDIDATE]: Term 1 election: Requested vote from peers 7532f3a035704530bff6b9605888f7c7 (127.0.98.65:33301), b4886a90d7904435ada30398c846f69e (127.0.98.66:39747)
I20260925 04:44:49.180555 8075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc0fd8a59f974f7593c5af33488baafd" candidate_uuid: "7d5cb59e32a844c988bd1754fa8f4550" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7532f3a035704530bff6b9605888f7c7"
I20260925 04:44:49.180682 8075 raft_consensus.cc:3037] T bc0fd8a59f974f7593c5af33488baafd P 7532f3a035704530bff6b9605888f7c7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:44:49.180953 8215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc0fd8a59f974f7593c5af33488baafd" candidate_uuid: "7d5cb59e32a844c988bd1754fa8f4550" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4886a90d7904435ada30398c846f69e"
I20260925 04:44:49.181089 8215 raft_consensus.cc:3037] T bc0fd8a59f974f7593c5af33488baafd P b4886a90d7904435ada30398c846f69e [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:44:49.181922 8075 raft_consensus.cc:2454] T bc0fd8a59f974f7593c5af33488baafd P 7532f3a035704530bff6b9605888f7c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d5cb59e32a844c988bd1754fa8f4550 in term 1.
I20260925 04:44:49.182287 8289 leader_election.cc:302] T bc0fd8a59f974f7593c5af33488baafd P 7d5cb59e32a844c988bd1754fa8f4550 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7532f3a035704530bff6b9605888f7c7, 7d5cb59e32a844c988bd1754fa8f4550; no voters:
I20260925 04:44:49.182430 8215 raft_consensus.cc:2454] T bc0fd8a59f974f7593c5af33488baafd P b4886a90d7904435ada30398c846f69e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d5cb59e32a844c988bd1754fa8f4550 in term 1.
I20260925 04:44:49.182498 8435 raft_consensus.cc:2781] T bc0fd8a59f974f7593c5af33488baafd P 7d5cb59e32a844c988bd1754fa8f4550 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:44:49.182585 8435 raft_consensus.cc:686] T bc0fd8a59f974f7593c5af33488baafd P 7d5cb59e32a844c988bd1754fa8f4550 [term 1 LEADER]: Becoming Leader. State: Replica: 7d5cb59e32a844c988bd1754fa8f4550, State: Running, Role: LEADER
I20260925 04:44:49.182742 8435 consensus_queue.cc:237] T bc0fd8a59f974f7593c5af33488baafd P 7d5cb59e32a844c988bd1754fa8f4550 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7532f3a035704530bff6b9605888f7c7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33301 } } peers { permanent_uuid: "b4886a90d7904435ada30398c846f69e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39747 } } peers { permanent_uuid: "7d5cb59e32a844c988bd1754fa8f4550" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 41117 } }
I20260925 04:44:49.183419 8439 tablet_replica.cc:439] T bc0fd8a59f974f7593c5af33488baafd P 7d5cb59e32a844c988bd1754fa8f4550: TxnStatusTablet state changed. Reason: New leader 7d5cb59e32a844c988bd1754fa8f4550. Latest consensus state: current_term: 1 leader_uuid: "7d5cb59e32a844c988bd1754fa8f4550" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7532f3a035704530bff6b9605888f7c7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33301 } } peers { permanent_uuid: "b4886a90d7904435ada30398c846f69e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39747 } } peers { permanent_uuid: "7d5cb59e32a844c988bd1754fa8f4550" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 41117 } } }
I20260925 04:44:49.183538 8439 tablet_replica.cc:442] T bc0fd8a59f974f7593c5af33488baafd P 7d5cb59e32a844c988bd1754fa8f4550: This TxnStatusTablet replica's current role is: LEADER
I20260925 04:44:49.183825 8469 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 04:44:49.183995 8469 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 04:44:49.184542 7917 catalog_manager.cc:5495] T bc0fd8a59f974f7593c5af33488baafd P 7d5cb59e32a844c988bd1754fa8f4550 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7d5cb59e32a844c988bd1754fa8f4550 (127.0.98.67). New cstate: current_term: 1 leader_uuid: "7d5cb59e32a844c988bd1754fa8f4550" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7532f3a035704530bff6b9605888f7c7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4886a90d7904435ada30398c846f69e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d5cb59e32a844c988bd1754fa8f4550" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 41117 } health_report { overall_health: HEALTHY } } }
I20260925 04:44:49.222220 8427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:49.218567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36826 (local address 127.0.98.67:41117)
0925 04:44:49.218682 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:49.218686 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:49.218707 (+ 21us) server_negotiation.cc:407] Connection header received
0925 04:44:49.218900 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:49.218903 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:49.218950 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:49.219066 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:44:49.219333 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:49.219919 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:49.221212 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:49.221420 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:49.221503 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:49.221813 (+ 310us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:44:49.221883 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:44:49.221937 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:49.222064 (+ 127us) server_negotiation.cc:299] Negotiation successful
0925 04:44:49.222109 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 04:44:49.222291 8464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:49.218430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:41117 (local address 127.0.0.1:36826)
0925 04:44:49.218605 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:49.218623 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:49.218730 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:49.219104 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:49.219107 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:49.219115 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 04:44:49.219225 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:49.219231 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:49.220071 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:49.220074 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:49.221075 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:49.221086 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:49.221528 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:49.221532 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:49.221620 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:49.221674 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 04:44:49.221963 (+ 289us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 04:44:49.221976 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 04:44:49.222043 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 04:44:49.222126 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260925 04:44:49.224864 8215 raft_consensus.cc:1261] T bc0fd8a59f974f7593c5af33488baafd P b4886a90d7904435ada30398c846f69e [term 1 FOLLOWER]: Refusing update from remote peer 7d5cb59e32a844c988bd1754fa8f4550: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:44:49.224938 8075 raft_consensus.cc:1261] T bc0fd8a59f974f7593c5af33488baafd P 7532f3a035704530bff6b9605888f7c7 [term 1 FOLLOWER]: Refusing update from remote peer 7d5cb59e32a844c988bd1754fa8f4550: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:44:49.225389 8435 consensus_queue.cc:1035] T bc0fd8a59f974f7593c5af33488baafd P 7d5cb59e32a844c988bd1754fa8f4550 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7532f3a035704530bff6b9605888f7c7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:44:49.225575 8439 consensus_queue.cc:1035] T bc0fd8a59f974f7593c5af33488baafd P 7d5cb59e32a844c988bd1754fa8f4550 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4886a90d7904435ada30398c846f69e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:44:49.227314 8436 tablet_replica.cc:439] T bc0fd8a59f974f7593c5af33488baafd P 7532f3a035704530bff6b9605888f7c7: TxnStatusTablet state changed. Reason: New leader 7d5cb59e32a844c988bd1754fa8f4550. Latest consensus state: current_term: 1 leader_uuid: "7d5cb59e32a844c988bd1754fa8f4550" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7532f3a035704530bff6b9605888f7c7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33301 } } peers { permanent_uuid: "b4886a90d7904435ada30398c846f69e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39747 } } peers { permanent_uuid: "7d5cb59e32a844c988bd1754fa8f4550" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 41117 } } }
I20260925 04:44:49.227438 8436 tablet_replica.cc:442] T bc0fd8a59f974f7593c5af33488baafd P 7532f3a035704530bff6b9605888f7c7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 04:44:49.227415 8437 tablet_replica.cc:439] T bc0fd8a59f974f7593c5af33488baafd P b4886a90d7904435ada30398c846f69e: TxnStatusTablet state changed. Reason: New leader 7d5cb59e32a844c988bd1754fa8f4550. Latest consensus state: current_term: 1 leader_uuid: "7d5cb59e32a844c988bd1754fa8f4550" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7532f3a035704530bff6b9605888f7c7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33301 } } peers { permanent_uuid: "b4886a90d7904435ada30398c846f69e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39747 } } peers { permanent_uuid: "7d5cb59e32a844c988bd1754fa8f4550" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 41117 } } }
I20260925 04:44:49.227535 8437 tablet_replica.cc:442] T bc0fd8a59f974f7593c5af33488baafd P b4886a90d7904435ada30398c846f69e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 04:44:49.229202 8439 tablet_replica.cc:439] T bc0fd8a59f974f7593c5af33488baafd P 7d5cb59e32a844c988bd1754fa8f4550: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7d5cb59e32a844c988bd1754fa8f4550" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7532f3a035704530bff6b9605888f7c7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33301 } } peers { permanent_uuid: "b4886a90d7904435ada30398c846f69e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39747 } } peers { permanent_uuid: "7d5cb59e32a844c988bd1754fa8f4550" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 41117 } } }
I20260925 04:44:49.229323 8439 tablet_replica.cc:442] T bc0fd8a59f974f7593c5af33488baafd P 7d5cb59e32a844c988bd1754fa8f4550: This TxnStatusTablet replica's current role is: LEADER
I20260925 04:44:49.229449 8439 tablet_replica.cc:439] T bc0fd8a59f974f7593c5af33488baafd P 7d5cb59e32a844c988bd1754fa8f4550: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7d5cb59e32a844c988bd1754fa8f4550" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7532f3a035704530bff6b9605888f7c7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33301 } } peers { permanent_uuid: "b4886a90d7904435ada30398c846f69e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39747 } } peers { permanent_uuid: "7d5cb59e32a844c988bd1754fa8f4550" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 41117 } } }
I20260925 04:44:49.229540 8439 tablet_replica.cc:442] T bc0fd8a59f974f7593c5af33488baafd P 7d5cb59e32a844c988bd1754fa8f4550: This TxnStatusTablet replica's current role is: LEADER
I20260925 04:44:49.231019 8436 tablet_replica.cc:439] T bc0fd8a59f974f7593c5af33488baafd P 7532f3a035704530bff6b9605888f7c7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7d5cb59e32a844c988bd1754fa8f4550" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7532f3a035704530bff6b9605888f7c7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33301 } } peers { permanent_uuid: "b4886a90d7904435ada30398c846f69e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39747 } } peers { permanent_uuid: "7d5cb59e32a844c988bd1754fa8f4550" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 41117 } } }
I20260925 04:44:49.231134 8436 tablet_replica.cc:442] T bc0fd8a59f974f7593c5af33488baafd P 7532f3a035704530bff6b9605888f7c7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 04:44:49.233848 8437 tablet_replica.cc:439] T bc0fd8a59f974f7593c5af33488baafd P b4886a90d7904435ada30398c846f69e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7d5cb59e32a844c988bd1754fa8f4550" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7532f3a035704530bff6b9605888f7c7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33301 } } peers { permanent_uuid: "b4886a90d7904435ada30398c846f69e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39747 } } peers { permanent_uuid: "7d5cb59e32a844c988bd1754fa8f4550" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 41117 } } }
I20260925 04:44:49.233974 8437 tablet_replica.cc:442] T bc0fd8a59f974f7593c5af33488baafd P b4886a90d7904435ada30398c846f69e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 04:44:49.245114 8427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:49.241314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:57039 (local address 127.0.98.67:41117)
0925 04:44:49.241462 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:49.241466 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:49.241479 (+ 13us) server_negotiation.cc:407] Connection header received
0925 04:44:49.241604 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:49.241607 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:49.241648 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:49.241710 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:44:49.241969 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:49.242666 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:49.243862 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:49.244194 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:49.244281 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:49.244608 (+ 327us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:44:49.244688 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:44:49.244757 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:49.244904 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 04:44:49.244985 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260925 04:44:49.245118 8407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:49.241234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:41117 (local address 127.0.98.67:57039)
0925 04:44:49.241394 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:49.241409 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:49.241511 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:49.241736 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:49.241739 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:49.241746 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 04:44:49.241846 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:49.241851 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:49.242773 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:49.242776 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:49.243726 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:49.243737 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:49.244303 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:49.244307 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:49.244391 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:49.244450 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 04:44:49.244775 (+ 325us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 04:44:49.244781 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 04:44:49.244848 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 04:44:49.244935 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 04:44:49.252337 8215 raft_consensus.cc:1261] T 1b25a2924d674b11b8de554694498895 P b4886a90d7904435ada30398c846f69e [term 1 FOLLOWER]: Refusing update from remote peer 7d5cb59e32a844c988bd1754fa8f4550: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:44:49.252599 8075 raft_consensus.cc:1261] T 1b25a2924d674b11b8de554694498895 P 7532f3a035704530bff6b9605888f7c7 [term 1 FOLLOWER]: Refusing update from remote peer 7d5cb59e32a844c988bd1754fa8f4550: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:44:49.252954 8435 consensus_queue.cc:1035] T 1b25a2924d674b11b8de554694498895 P 7d5cb59e32a844c988bd1754fa8f4550 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7532f3a035704530bff6b9605888f7c7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:44:49.253123 8448 consensus_queue.cc:1035] T 1b25a2924d674b11b8de554694498895 P 7d5cb59e32a844c988bd1754fa8f4550 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4886a90d7904435ada30398c846f69e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:44:49.257298 8442 mvcc.cc:205] Tried to move back new op lower bound from 7333115859974361088 to 7333115858869182464. Current Snapshot: MvccSnapshot[applied={T|T < 7333115859974361088}]
I20260925 04:44:49.305759 7917 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35364:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:44:49.306001 7917 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35364:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:44:49.308575 7917 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1f45ce55bb3f415c8d7178965484a569: Sending DeleteTablet for 3 replicas of tablet 1b25a2924d674b11b8de554694498895
I20260925 04:44:49.309347 8195 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b25a2924d674b11b8de554694498895 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:44:49 UTC) from {username='oryx', principal='oryx/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:34876
I20260925 04:44:49.309450 8334 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b25a2924d674b11b8de554694498895 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:44:49 UTC) from {username='oryx', principal='oryx/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:36804
I20260925 04:44:49.309576 8453 tablet_replica.cc:330] T 1b25a2924d674b11b8de554694498895 P b4886a90d7904435ada30398c846f69e: stopping tablet replica
I20260925 04:44:49.309584 8055 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b25a2924d674b11b8de554694498895 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:44:49 UTC) from {username='oryx', principal='oryx/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:41128
I20260925 04:44:49.309744 8453 raft_consensus.cc:2229] T 1b25a2924d674b11b8de554694498895 P b4886a90d7904435ada30398c846f69e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:44:49.309798 8452 tablet_replica.cc:330] T 1b25a2924d674b11b8de554694498895 P 7d5cb59e32a844c988bd1754fa8f4550: stopping tablet replica
I20260925 04:44:49.309800 8454 tablet_replica.cc:330] T 1b25a2924d674b11b8de554694498895 P 7532f3a035704530bff6b9605888f7c7: stopping tablet replica
I20260925 04:44:49.309883 8453 raft_consensus.cc:2258] T 1b25a2924d674b11b8de554694498895 P b4886a90d7904435ada30398c846f69e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:44:49.309921 8452 raft_consensus.cc:2229] T 1b25a2924d674b11b8de554694498895 P 7d5cb59e32a844c988bd1754fa8f4550 [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:44:49.309923 8454 raft_consensus.cc:2229] T 1b25a2924d674b11b8de554694498895 P 7532f3a035704530bff6b9605888f7c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:44:49.310038 8454 raft_consensus.cc:2258] T 1b25a2924d674b11b8de554694498895 P 7532f3a035704530bff6b9605888f7c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:44:49.310114 8452 raft_consensus.cc:2258] T 1b25a2924d674b11b8de554694498895 P 7d5cb59e32a844c988bd1754fa8f4550 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:44:49.311070 8453 ts_tablet_manager.cc:1905] T 1b25a2924d674b11b8de554694498895 P b4886a90d7904435ada30398c846f69e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:44:49.311151 8452 ts_tablet_manager.cc:1905] T 1b25a2924d674b11b8de554694498895 P 7d5cb59e32a844c988bd1754fa8f4550: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:44:49.314383 8452 ts_tablet_manager.cc:1918] T 1b25a2924d674b11b8de554694498895 P 7d5cb59e32a844c988bd1754fa8f4550: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 04:44:49.314474 8452 log.cc:1196] T 1b25a2924d674b11b8de554694498895 P 7d5cb59e32a844c988bd1754fa8f4550: Deleting WAL directory at /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/minicluster-data/ts-2/wal/wals/1b25a2924d674b11b8de554694498895
I20260925 04:44:49.314793 8452 ts_tablet_manager.cc:1939] T 1b25a2924d674b11b8de554694498895 P 7d5cb59e32a844c988bd1754fa8f4550: Deleting consensus metadata
I20260925 04:44:49.314918 8453 ts_tablet_manager.cc:1918] T 1b25a2924d674b11b8de554694498895 P b4886a90d7904435ada30398c846f69e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 04:44:49.314997 8453 log.cc:1196] T 1b25a2924d674b11b8de554694498895 P b4886a90d7904435ada30398c846f69e: Deleting WAL directory at /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/minicluster-data/ts-1/wal/wals/1b25a2924d674b11b8de554694498895
I20260925 04:44:49.315286 8453 ts_tablet_manager.cc:1939] T 1b25a2924d674b11b8de554694498895 P b4886a90d7904435ada30398c846f69e: Deleting consensus metadata
I20260925 04:44:49.315624 7903 catalog_manager.cc:4841] TS 7d5cb59e32a844c988bd1754fa8f4550 (127.0.98.67:41117): tablet 1b25a2924d674b11b8de554694498895 (table test-table [id=f12b58e6b4254609838581aed53b1c02]) successfully deleted
I20260925 04:44:49.316183 7902 catalog_manager.cc:4841] TS b4886a90d7904435ada30398c846f69e (127.0.98.66:39747): tablet 1b25a2924d674b11b8de554694498895 (table test-table [id=f12b58e6b4254609838581aed53b1c02]) successfully deleted
I20260925 04:44:49.316725 8454 ts_tablet_manager.cc:1905] T 1b25a2924d674b11b8de554694498895 P 7532f3a035704530bff6b9605888f7c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:44:49.320282 8454 ts_tablet_manager.cc:1918] T 1b25a2924d674b11b8de554694498895 P 7532f3a035704530bff6b9605888f7c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 04:44:49.320398 8454 log.cc:1196] T 1b25a2924d674b11b8de554694498895 P 7532f3a035704530bff6b9605888f7c7: Deleting WAL directory at /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/minicluster-data/ts-0/wal/wals/1b25a2924d674b11b8de554694498895
I20260925 04:44:49.320633 8454 ts_tablet_manager.cc:1939] T 1b25a2924d674b11b8de554694498895 P 7532f3a035704530bff6b9605888f7c7: Deleting consensus metadata
I20260925 04:44:49.321354 7905 catalog_manager.cc:4841] TS 7532f3a035704530bff6b9605888f7c7 (127.0.98.65:33301): tablet 1b25a2924d674b11b8de554694498895 (table test-table [id=f12b58e6b4254609838581aed53b1c02]) successfully deleted
Sep 25 04:44:49 dist-test-slave-kghz krb5kdc[7873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 04:44:49 dist-test-slave-kghz krb5kdc[7873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:44:49 dist-test-slave-kghz krb5kdc[7873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
W20260925 04:44:49.343660 7967 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:44:49.336066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51050 (local address 127.0.98.126:40947)
0925 04:44:49.336227 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:49.336233 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:49.336385 (+ 152us) server_negotiation.cc:407] Connection header received
0925 04:44:49.336535 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:49.336538 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:49.336592 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:49.336695 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:49.337428 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:49.338406 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:49.339424 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:49.339801 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:49.339943 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:49.343519 (+ 3576us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51050: BlockingRecv error: recv got EOF from 127.0.0.1:51050 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 25 04:44:49 dist-test-slave-kghz krb5kdc[7873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM
I20260925 04:44:49.354354 7967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:49.345145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51052 (local address 127.0.98.126:40947)
0925 04:44:49.345254 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:49.345258 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:49.345333 (+ 75us) server_negotiation.cc:407] Connection header received
0925 04:44:49.345424 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:49.345427 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:49.345475 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:49.345575 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:49.346201 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:49.347124 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:49.347935 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:49.348296 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:49.348441 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:49.351090 (+ 2649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:49.351093 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:49.351095 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:49.351119 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:49.352556 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:49.353045 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:49.353048 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:49.353049 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:49.353126 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:49.353404 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:49.353406 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:49.353408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:49.353575 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:49.353713 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:49.354047 (+ 334us) server_negotiation.cc:299] Negotiation successful
0925 04:44:49.354142 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 04:44:49.357216 7917 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51052:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:44:49.357708 7917 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:44:49.362581 8055 tablet_service.cc:1467] Processing CreateTablet for tablet 1fc38b45c0354a7ba6f644d7434d64a2 (DEFAULT_TABLE table=test-table [id=ca79134870244030b6d2a7c7e0a3fd3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:44:49.362664 8195 tablet_service.cc:1467] Processing CreateTablet for tablet 1fc38b45c0354a7ba6f644d7434d64a2 (DEFAULT_TABLE table=test-table [id=ca79134870244030b6d2a7c7e0a3fd3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:44:49.362902 8055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fc38b45c0354a7ba6f644d7434d64a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:44:49.362847 8334 tablet_service.cc:1467] Processing CreateTablet for tablet 1fc38b45c0354a7ba6f644d7434d64a2 (DEFAULT_TABLE table=test-table [id=ca79134870244030b6d2a7c7e0a3fd3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:44:49.362979 8195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fc38b45c0354a7ba6f644d7434d64a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:44:49.363101 8334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fc38b45c0354a7ba6f644d7434d64a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:44:49.364967 8431 tablet_bootstrap.cc:492] T 1fc38b45c0354a7ba6f644d7434d64a2 P b4886a90d7904435ada30398c846f69e: Bootstrap starting.
I20260925 04:44:49.365316 8429 tablet_bootstrap.cc:492] T 1fc38b45c0354a7ba6f644d7434d64a2 P 7d5cb59e32a844c988bd1754fa8f4550: Bootstrap starting.
I20260925 04:44:49.365949 8431 tablet_bootstrap.cc:654] T 1fc38b45c0354a7ba6f644d7434d64a2 P b4886a90d7904435ada30398c846f69e: Neither blocks nor log segments found. Creating new log.
I20260925 04:44:49.366137 8429 tablet_bootstrap.cc:654] T 1fc38b45c0354a7ba6f644d7434d64a2 P 7d5cb59e32a844c988bd1754fa8f4550: Neither blocks nor log segments found. Creating new log.
I20260925 04:44:49.366652 8430 tablet_bootstrap.cc:492] T 1fc38b45c0354a7ba6f644d7434d64a2 P 7532f3a035704530bff6b9605888f7c7: Bootstrap starting.
I20260925 04:44:49.367218 8431 tablet_bootstrap.cc:492] T 1fc38b45c0354a7ba6f644d7434d64a2 P b4886a90d7904435ada30398c846f69e: No bootstrap required, opened a new log
I20260925 04:44:49.367311 8431 ts_tablet_manager.cc:1397] T 1fc38b45c0354a7ba6f644d7434d64a2 P b4886a90d7904435ada30398c846f69e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260925 04:44:49.367430 8429 tablet_bootstrap.cc:492] T 1fc38b45c0354a7ba6f644d7434d64a2 P 7d5cb59e32a844c988bd1754fa8f4550: No bootstrap required, opened a new log
I20260925 04:44:49.367512 8429 ts_tablet_manager.cc:1397] T 1fc38b45c0354a7ba6f644d7434d64a2 P 7d5cb59e32a844c988bd1754fa8f4550: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 04:44:49.367559 8430 tablet_bootstrap.cc:654] T 1fc38b45c0354a7ba6f644d7434d64a2 P 7532f3a035704530bff6b9605888f7c7: Neither blocks nor log segments found. Creating new log.
I20260925 04:44:49.367689 8431 raft_consensus.cc:348] T 1fc38b45c0354a7ba6f644d7434d64a2 P b4886a90d7904435ada30398c846f69e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7532f3a035704530bff6b9605888f7c7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33301 } } peers { permanent_uuid: "b4886a90d7904435ada30398c846f69e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39747 } } peers { permanent_uuid: "7d5cb59e32a844c988bd1754fa8f4550" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 41117 } }
I20260925 04:44:49.367771 8431 raft_consensus.cc:374] T 1fc38b45c0354a7ba6f644d7434d64a2 P b4886a90d7904435ada30398c846f69e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:44:49.367792 8431 raft_consensus.cc:729] T 1fc38b45c0354a7ba6f644d7434d64a2 P b4886a90d7904435ada30398c846f69e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4886a90d7904435ada30398c846f69e, State: Initialized, Role: FOLLOWER
I20260925 04:44:49.367923 8431 consensus_queue.cc:260] T 1fc38b45c0354a7ba6f644d7434d64a2 P b4886a90d7904435ada30398c846f69e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7532f3a035704530bff6b9605888f7c7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33301 } } peers { permanent_uuid: "b4886a90d7904435ada30398c846f69e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39747 } } peers { permanent_uuid: "7d5cb59e32a844c988bd1754fa8f4550" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 41117 } }
I20260925 04:44:49.367930 8429 raft_consensus.cc:348] T 1fc38b45c0354a7ba6f644d7434d64a2 P 7d5cb59e32a844c988bd1754fa8f4550 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7532f3a035704530bff6b9605888f7c7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33301 } } peers { permanent_uuid: "b4886a90d7904435ada30398c846f69e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39747 } } peers { permanent_uuid: "7d5cb59e32a844c988bd1754fa8f4550" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 41117 } }
I20260925 04:44:49.368058 8429 raft_consensus.cc:374] T 1fc38b45c0354a7ba6f644d7434d64a2 P 7d5cb59e32a844c988bd1754fa8f4550 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:44:49.368093 8429 raft_consensus.cc:729] T 1fc38b45c0354a7ba6f644d7434d64a2 P 7d5cb59e32a844c988bd1754fa8f4550 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d5cb59e32a844c988bd1754fa8f4550, State: Initialized, Role: FOLLOWER
I20260925 04:44:49.368220 8431 ts_tablet_manager.cc:1428] T 1fc38b45c0354a7ba6f644d7434d64a2 P b4886a90d7904435ada30398c846f69e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 04:44:49.368211 8429 consensus_queue.cc:260] T 1fc38b45c0354a7ba6f644d7434d64a2 P 7d5cb59e32a844c988bd1754fa8f4550 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7532f3a035704530bff6b9605888f7c7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33301 } } peers { permanent_uuid: "b4886a90d7904435ada30398c846f69e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39747 } } peers { permanent_uuid: "7d5cb59e32a844c988bd1754fa8f4550" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 41117 } }
I20260925 04:44:49.368497 8429 ts_tablet_manager.cc:1428] T 1fc38b45c0354a7ba6f644d7434d64a2 P 7d5cb59e32a844c988bd1754fa8f4550: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 04:44:49.368968 8430 tablet_bootstrap.cc:492] T 1fc38b45c0354a7ba6f644d7434d64a2 P 7532f3a035704530bff6b9605888f7c7: No bootstrap required, opened a new log
I20260925 04:44:49.369073 8430 ts_tablet_manager.cc:1397] T 1fc38b45c0354a7ba6f644d7434d64a2 P 7532f3a035704530bff6b9605888f7c7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 04:44:49.369582 8430 raft_consensus.cc:348] T 1fc38b45c0354a7ba6f644d7434d64a2 P 7532f3a035704530bff6b9605888f7c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7532f3a035704530bff6b9605888f7c7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33301 } } peers { permanent_uuid: "b4886a90d7904435ada30398c846f69e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39747 } } peers { permanent_uuid: "7d5cb59e32a844c988bd1754fa8f4550" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 41117 } }
I20260925 04:44:49.369724 8430 raft_consensus.cc:374] T 1fc38b45c0354a7ba6f644d7434d64a2 P 7532f3a035704530bff6b9605888f7c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:44:49.369763 8430 raft_consensus.cc:729] T 1fc38b45c0354a7ba6f644d7434d64a2 P 7532f3a035704530bff6b9605888f7c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7532f3a035704530bff6b9605888f7c7, State: Initialized, Role: FOLLOWER
I20260925 04:44:49.369884 8430 consensus_queue.cc:260] T 1fc38b45c0354a7ba6f644d7434d64a2 P 7532f3a035704530bff6b9605888f7c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7532f3a035704530bff6b9605888f7c7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33301 } } peers { permanent_uuid: "b4886a90d7904435ada30398c846f69e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39747 } } peers { permanent_uuid: "7d5cb59e32a844c988bd1754fa8f4550" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 41117 } }
I20260925 04:44:49.370205 8430 ts_tablet_manager.cc:1428] T 1fc38b45c0354a7ba6f644d7434d64a2 P 7532f3a035704530bff6b9605888f7c7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 04:44:49.744889 8439 raft_consensus.cc:482] T 1fc38b45c0354a7ba6f644d7434d64a2 P 7d5cb59e32a844c988bd1754fa8f4550 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:44:49.745031 8439 raft_consensus.cc:504] T 1fc38b45c0354a7ba6f644d7434d64a2 P 7d5cb59e32a844c988bd1754fa8f4550 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7532f3a035704530bff6b9605888f7c7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33301 } } peers { permanent_uuid: "b4886a90d7904435ada30398c846f69e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39747 } } peers { permanent_uuid: "7d5cb59e32a844c988bd1754fa8f4550" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 41117 } }
I20260925 04:44:49.745558 8439 leader_election.cc:288] T 1fc38b45c0354a7ba6f644d7434d64a2 P 7d5cb59e32a844c988bd1754fa8f4550 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7532f3a035704530bff6b9605888f7c7 (127.0.98.65:33301), b4886a90d7904435ada30398c846f69e (127.0.98.66:39747)
I20260925 04:44:49.746003 8075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fc38b45c0354a7ba6f644d7434d64a2" candidate_uuid: "7d5cb59e32a844c988bd1754fa8f4550" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7532f3a035704530bff6b9605888f7c7" is_pre_election: true
I20260925 04:44:49.746048 8213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fc38b45c0354a7ba6f644d7434d64a2" candidate_uuid: "7d5cb59e32a844c988bd1754fa8f4550" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4886a90d7904435ada30398c846f69e" is_pre_election: true
I20260925 04:44:49.746160 8075 raft_consensus.cc:2454] T 1fc38b45c0354a7ba6f644d7434d64a2 P 7532f3a035704530bff6b9605888f7c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d5cb59e32a844c988bd1754fa8f4550 in term 0.
I20260925 04:44:49.746202 8213 raft_consensus.cc:2454] T 1fc38b45c0354a7ba6f644d7434d64a2 P b4886a90d7904435ada30398c846f69e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d5cb59e32a844c988bd1754fa8f4550 in term 0.
I20260925 04:44:49.746500 8289 leader_election.cc:302] T 1fc38b45c0354a7ba6f644d7434d64a2 P 7d5cb59e32a844c988bd1754fa8f4550 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7532f3a035704530bff6b9605888f7c7, 7d5cb59e32a844c988bd1754fa8f4550; no voters:
I20260925 04:44:49.746676 8439 raft_consensus.cc:2781] T 1fc38b45c0354a7ba6f644d7434d64a2 P 7d5cb59e32a844c988bd1754fa8f4550 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:44:49.746737 8439 raft_consensus.cc:482] T 1fc38b45c0354a7ba6f644d7434d64a2 P 7d5cb59e32a844c988bd1754fa8f4550 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:44:49.746784 8439 raft_consensus.cc:3037] T 1fc38b45c0354a7ba6f644d7434d64a2 P 7d5cb59e32a844c988bd1754fa8f4550 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:44:49.747754 8439 raft_consensus.cc:504] T 1fc38b45c0354a7ba6f644d7434d64a2 P 7d5cb59e32a844c988bd1754fa8f4550 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7532f3a035704530bff6b9605888f7c7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33301 } } peers { permanent_uuid: "b4886a90d7904435ada30398c846f69e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39747 } } peers { permanent_uuid: "7d5cb59e32a844c988bd1754fa8f4550" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 41117 } }
I20260925 04:44:49.748098 8439 leader_election.cc:288] T 1fc38b45c0354a7ba6f644d7434d64a2 P 7d5cb59e32a844c988bd1754fa8f4550 [CANDIDATE]: Term 1 election: Requested vote from peers 7532f3a035704530bff6b9605888f7c7 (127.0.98.65:33301), b4886a90d7904435ada30398c846f69e (127.0.98.66:39747)
I20260925 04:44:49.748405 8075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fc38b45c0354a7ba6f644d7434d64a2" candidate_uuid: "7d5cb59e32a844c988bd1754fa8f4550" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7532f3a035704530bff6b9605888f7c7"
I20260925 04:44:49.748467 8213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fc38b45c0354a7ba6f644d7434d64a2" candidate_uuid: "7d5cb59e32a844c988bd1754fa8f4550" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4886a90d7904435ada30398c846f69e"
I20260925 04:44:49.748528 8075 raft_consensus.cc:3037] T 1fc38b45c0354a7ba6f644d7434d64a2 P 7532f3a035704530bff6b9605888f7c7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:44:49.748574 8213 raft_consensus.cc:3037] T 1fc38b45c0354a7ba6f644d7434d64a2 P b4886a90d7904435ada30398c846f69e [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:44:49.749629 8075 raft_consensus.cc:2454] T 1fc38b45c0354a7ba6f644d7434d64a2 P 7532f3a035704530bff6b9605888f7c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d5cb59e32a844c988bd1754fa8f4550 in term 1.
I20260925 04:44:49.749629 8213 raft_consensus.cc:2454] T 1fc38b45c0354a7ba6f644d7434d64a2 P b4886a90d7904435ada30398c846f69e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d5cb59e32a844c988bd1754fa8f4550 in term 1.
I20260925 04:44:49.749902 8289 leader_election.cc:302] T 1fc38b45c0354a7ba6f644d7434d64a2 P 7d5cb59e32a844c988bd1754fa8f4550 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7532f3a035704530bff6b9605888f7c7, 7d5cb59e32a844c988bd1754fa8f4550; no voters:
I20260925 04:44:49.750048 8439 raft_consensus.cc:2781] T 1fc38b45c0354a7ba6f644d7434d64a2 P 7d5cb59e32a844c988bd1754fa8f4550 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:44:49.750137 8439 raft_consensus.cc:686] T 1fc38b45c0354a7ba6f644d7434d64a2 P 7d5cb59e32a844c988bd1754fa8f4550 [term 1 LEADER]: Becoming Leader. State: Replica: 7d5cb59e32a844c988bd1754fa8f4550, State: Running, Role: LEADER
I20260925 04:44:49.750300 8439 consensus_queue.cc:237] T 1fc38b45c0354a7ba6f644d7434d64a2 P 7d5cb59e32a844c988bd1754fa8f4550 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7532f3a035704530bff6b9605888f7c7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33301 } } peers { permanent_uuid: "b4886a90d7904435ada30398c846f69e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39747 } } peers { permanent_uuid: "7d5cb59e32a844c988bd1754fa8f4550" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 41117 } }
I20260925 04:44:49.751953 7917 catalog_manager.cc:5495] T 1fc38b45c0354a7ba6f644d7434d64a2 P 7d5cb59e32a844c988bd1754fa8f4550 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7d5cb59e32a844c988bd1754fa8f4550 (127.0.98.67). New cstate: current_term: 1 leader_uuid: "7d5cb59e32a844c988bd1754fa8f4550" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7532f3a035704530bff6b9605888f7c7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4886a90d7904435ada30398c846f69e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d5cb59e32a844c988bd1754fa8f4550" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 41117 } health_report { overall_health: HEALTHY } } }
I20260925 04:44:49.822069 8496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:49.818811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36982 (local address 127.0.98.67:41117)
0925 04:44:49.819084 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:49.819089 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:49.819102 (+ 13us) server_negotiation.cc:407] Connection header received
0925 04:44:49.819156 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:49.819158 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:49.819202 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:49.819293 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:44:49.819565 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:49.820092 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:49.820971 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:49.821206 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:49.821308 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:49.821569 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:44:49.821651 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:44:49.821730 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:49.821843 (+ 113us) server_negotiation.cc:299] Negotiation successful
0925 04:44:49.821900 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":83,"threads_started":1}
I20260925 04:44:49.824230 8075 raft_consensus.cc:1261] T 1fc38b45c0354a7ba6f644d7434d64a2 P 7532f3a035704530bff6b9605888f7c7 [term 1 FOLLOWER]: Refusing update from remote peer 7d5cb59e32a844c988bd1754fa8f4550: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:44:49.824240 8213 raft_consensus.cc:1261] T 1fc38b45c0354a7ba6f644d7434d64a2 P b4886a90d7904435ada30398c846f69e [term 1 FOLLOWER]: Refusing update from remote peer 7d5cb59e32a844c988bd1754fa8f4550: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:44:49.824704 8439 consensus_queue.cc:1035] T 1fc38b45c0354a7ba6f644d7434d64a2 P 7d5cb59e32a844c988bd1754fa8f4550 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7532f3a035704530bff6b9605888f7c7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:44:49.824859 8448 consensus_queue.cc:1035] T 1fc38b45c0354a7ba6f644d7434d64a2 P 7d5cb59e32a844c988bd1754fa8f4550 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4886a90d7904435ada30398c846f69e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:44:49.831722 7916 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51052:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:44:49.832034 7916 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51052:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:44:49.834357 7916 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1f45ce55bb3f415c8d7178965484a569: Sending DeleteTablet for 3 replicas of tablet 1fc38b45c0354a7ba6f644d7434d64a2
I20260925 04:44:49.834885 8055 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fc38b45c0354a7ba6f644d7434d64a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:44:49 UTC) from {username='oryx', principal='oryx/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:41128
I20260925 04:44:49.835041 8195 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fc38b45c0354a7ba6f644d7434d64a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:44:49 UTC) from {username='oryx', principal='oryx/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:34876
I20260925 04:44:49.835214 8506 tablet_replica.cc:330] T 1fc38b45c0354a7ba6f644d7434d64a2 P 7532f3a035704530bff6b9605888f7c7: stopping tablet replica
I20260925 04:44:49.835175 8334 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fc38b45c0354a7ba6f644d7434d64a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:44:49 UTC) from {username='oryx', principal='oryx/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:36804
I20260925 04:44:49.835322 8506 raft_consensus.cc:2229] T 1fc38b45c0354a7ba6f644d7434d64a2 P 7532f3a035704530bff6b9605888f7c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:44:49.835400 8507 tablet_replica.cc:330] T 1fc38b45c0354a7ba6f644d7434d64a2 P b4886a90d7904435ada30398c846f69e: stopping tablet replica
I20260925 04:44:49.835451 8506 raft_consensus.cc:2258] T 1fc38b45c0354a7ba6f644d7434d64a2 P 7532f3a035704530bff6b9605888f7c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:44:49.835518 8507 raft_consensus.cc:2229] T 1fc38b45c0354a7ba6f644d7434d64a2 P b4886a90d7904435ada30398c846f69e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:44:49.835661 8507 raft_consensus.cc:2258] T 1fc38b45c0354a7ba6f644d7434d64a2 P b4886a90d7904435ada30398c846f69e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:44:49.835714 7916 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51052:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 04:44:49.835923 8508 tablet_replica.cc:330] T 1fc38b45c0354a7ba6f644d7434d64a2 P 7d5cb59e32a844c988bd1754fa8f4550: stopping tablet replica
I20260925 04:44:49.836061 8508 raft_consensus.cc:2229] T 1fc38b45c0354a7ba6f644d7434d64a2 P 7d5cb59e32a844c988bd1754fa8f4550 [term 1 LEADER]: Raft consensus shutting down.
W20260925 04:44:49.836203 7916 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:44:49.836259 8508 raft_consensus.cc:2258] T 1fc38b45c0354a7ba6f644d7434d64a2 P 7d5cb59e32a844c988bd1754fa8f4550 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:44:49.836419 8507 ts_tablet_manager.cc:1905] T 1fc38b45c0354a7ba6f644d7434d64a2 P b4886a90d7904435ada30398c846f69e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:44:49.836828 8508 ts_tablet_manager.cc:1905] T 1fc38b45c0354a7ba6f644d7434d64a2 P 7d5cb59e32a844c988bd1754fa8f4550: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:44:49.837463 8506 ts_tablet_manager.cc:1905] T 1fc38b45c0354a7ba6f644d7434d64a2 P 7532f3a035704530bff6b9605888f7c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:44:49.839674 8508 ts_tablet_manager.cc:1918] T 1fc38b45c0354a7ba6f644d7434d64a2 P 7d5cb59e32a844c988bd1754fa8f4550: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:44:49.839751 8508 log.cc:1196] T 1fc38b45c0354a7ba6f644d7434d64a2 P 7d5cb59e32a844c988bd1754fa8f4550: Deleting WAL directory at /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/minicluster-data/ts-2/wal/wals/1fc38b45c0354a7ba6f644d7434d64a2
I20260925 04:44:49.839857 8507 ts_tablet_manager.cc:1918] T 1fc38b45c0354a7ba6f644d7434d64a2 P b4886a90d7904435ada30398c846f69e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:44:49.839948 8507 log.cc:1196] T 1fc38b45c0354a7ba6f644d7434d64a2 P b4886a90d7904435ada30398c846f69e: Deleting WAL directory at /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/minicluster-data/ts-1/wal/wals/1fc38b45c0354a7ba6f644d7434d64a2
I20260925 04:44:49.840022 8508 ts_tablet_manager.cc:1939] T 1fc38b45c0354a7ba6f644d7434d64a2 P 7d5cb59e32a844c988bd1754fa8f4550: Deleting consensus metadata
I20260925 04:44:49.840232 8507 ts_tablet_manager.cc:1939] T 1fc38b45c0354a7ba6f644d7434d64a2 P b4886a90d7904435ada30398c846f69e: Deleting consensus metadata
I20260925 04:44:49.840289 8506 ts_tablet_manager.cc:1918] T 1fc38b45c0354a7ba6f644d7434d64a2 P 7532f3a035704530bff6b9605888f7c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:44:49.840358 8506 log.cc:1196] T 1fc38b45c0354a7ba6f644d7434d64a2 P 7532f3a035704530bff6b9605888f7c7: Deleting WAL directory at /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/minicluster-data/ts-0/wal/wals/1fc38b45c0354a7ba6f644d7434d64a2
I20260925 04:44:49.840624 8506 ts_tablet_manager.cc:1939] T 1fc38b45c0354a7ba6f644d7434d64a2 P 7532f3a035704530bff6b9605888f7c7: Deleting consensus metadata
I20260925 04:44:49.840770 7903 catalog_manager.cc:4841] TS 7d5cb59e32a844c988bd1754fa8f4550 (127.0.98.67:41117): tablet 1fc38b45c0354a7ba6f644d7434d64a2 (table test-table [id=ca79134870244030b6d2a7c7e0a3fd3e]) successfully deleted
I20260925 04:44:49.841029 7902 catalog_manager.cc:4841] TS b4886a90d7904435ada30398c846f69e (127.0.98.66:39747): tablet 1fc38b45c0354a7ba6f644d7434d64a2 (table test-table [id=ca79134870244030b6d2a7c7e0a3fd3e]) successfully deleted
I20260925 04:44:49.841408 7905 catalog_manager.cc:4841] TS 7532f3a035704530bff6b9605888f7c7 (127.0.98.65:33301): tablet 1fc38b45c0354a7ba6f644d7434d64a2 (table test-table [id=ca79134870244030b6d2a7c7e0a3fd3e]) successfully deleted
I20260925 04:44:49.843078 8055 tablet_service.cc:1467] Processing CreateTablet for tablet d896951b99bf46259fc1f05310408b13 (DEFAULT_TABLE table=test-table [id=8bcf615b5718407c8a6a71fc4fa2d592]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:44:49.843115 8195 tablet_service.cc:1467] Processing CreateTablet for tablet d896951b99bf46259fc1f05310408b13 (DEFAULT_TABLE table=test-table [id=8bcf615b5718407c8a6a71fc4fa2d592]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:44:49.843336 8055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d896951b99bf46259fc1f05310408b13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:44:49.843406 8195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d896951b99bf46259fc1f05310408b13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:44:49.845453 8430 tablet_bootstrap.cc:492] T d896951b99bf46259fc1f05310408b13 P 7532f3a035704530bff6b9605888f7c7: Bootstrap starting.
I20260925 04:44:49.845571 8431 tablet_bootstrap.cc:492] T d896951b99bf46259fc1f05310408b13 P b4886a90d7904435ada30398c846f69e: Bootstrap starting.
I20260925 04:44:49.846504 8431 tablet_bootstrap.cc:654] T d896951b99bf46259fc1f05310408b13 P b4886a90d7904435ada30398c846f69e: Neither blocks nor log segments found. Creating new log.
I20260925 04:44:49.846504 8430 tablet_bootstrap.cc:654] T d896951b99bf46259fc1f05310408b13 P 7532f3a035704530bff6b9605888f7c7: Neither blocks nor log segments found. Creating new log.
I20260925 04:44:49.846767 8334 tablet_service.cc:1467] Processing CreateTablet for tablet d896951b99bf46259fc1f05310408b13 (DEFAULT_TABLE table=test-table [id=8bcf615b5718407c8a6a71fc4fa2d592]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:44:49.847075 8334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d896951b99bf46259fc1f05310408b13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:44:49.847730 8430 tablet_bootstrap.cc:492] T d896951b99bf46259fc1f05310408b13 P 7532f3a035704530bff6b9605888f7c7: No bootstrap required, opened a new log
I20260925 04:44:49.847823 8430 ts_tablet_manager.cc:1397] T d896951b99bf46259fc1f05310408b13 P 7532f3a035704530bff6b9605888f7c7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260925 04:44:49.847986 8431 tablet_bootstrap.cc:492] T d896951b99bf46259fc1f05310408b13 P b4886a90d7904435ada30398c846f69e: No bootstrap required, opened a new log
I20260925 04:44:49.848074 8431 ts_tablet_manager.cc:1397] T d896951b99bf46259fc1f05310408b13 P b4886a90d7904435ada30398c846f69e: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260925 04:44:49.848287 8430 raft_consensus.cc:348] T d896951b99bf46259fc1f05310408b13 P 7532f3a035704530bff6b9605888f7c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7532f3a035704530bff6b9605888f7c7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33301 } } peers { permanent_uuid: "b4886a90d7904435ada30398c846f69e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39747 } } peers { permanent_uuid: "7d5cb59e32a844c988bd1754fa8f4550" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 41117 } }
I20260925 04:44:49.848402 8430 raft_consensus.cc:374] T d896951b99bf46259fc1f05310408b13 P 7532f3a035704530bff6b9605888f7c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:44:49.848435 8430 raft_consensus.cc:729] T d896951b99bf46259fc1f05310408b13 P 7532f3a035704530bff6b9605888f7c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7532f3a035704530bff6b9605888f7c7, State: Initialized, Role: FOLLOWER
I20260925 04:44:49.848495 8431 raft_consensus.cc:348] T d896951b99bf46259fc1f05310408b13 P b4886a90d7904435ada30398c846f69e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7532f3a035704530bff6b9605888f7c7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33301 } } peers { permanent_uuid: "b4886a90d7904435ada30398c846f69e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39747 } } peers { permanent_uuid: "7d5cb59e32a844c988bd1754fa8f4550" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 41117 } }
I20260925 04:44:49.848598 8431 raft_consensus.cc:374] T d896951b99bf46259fc1f05310408b13 P b4886a90d7904435ada30398c846f69e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:44:49.848577 8430 consensus_queue.cc:260] T d896951b99bf46259fc1f05310408b13 P 7532f3a035704530bff6b9605888f7c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7532f3a035704530bff6b9605888f7c7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33301 } } peers { permanent_uuid: "b4886a90d7904435ada30398c846f69e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39747 } } peers { permanent_uuid: "7d5cb59e32a844c988bd1754fa8f4550" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 41117 } }
I20260925 04:44:49.848639 8431 raft_consensus.cc:729] T d896951b99bf46259fc1f05310408b13 P b4886a90d7904435ada30398c846f69e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4886a90d7904435ada30398c846f69e, State: Initialized, Role: FOLLOWER
I20260925 04:44:49.848752 8431 consensus_queue.cc:260] T d896951b99bf46259fc1f05310408b13 P b4886a90d7904435ada30398c846f69e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7532f3a035704530bff6b9605888f7c7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33301 } } peers { permanent_uuid: "b4886a90d7904435ada30398c846f69e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39747 } } peers { permanent_uuid: "7d5cb59e32a844c988bd1754fa8f4550" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 41117 } }
I20260925 04:44:49.848992 8431 ts_tablet_manager.cc:1428] T d896951b99bf46259fc1f05310408b13 P b4886a90d7904435ada30398c846f69e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 04:44:49.849442 8430 ts_tablet_manager.cc:1428] T d896951b99bf46259fc1f05310408b13 P 7532f3a035704530bff6b9605888f7c7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260925 04:44:49.849553 8429 tablet_bootstrap.cc:492] T d896951b99bf46259fc1f05310408b13 P 7d5cb59e32a844c988bd1754fa8f4550: Bootstrap starting.
I20260925 04:44:49.850482 8429 tablet_bootstrap.cc:654] T d896951b99bf46259fc1f05310408b13 P 7d5cb59e32a844c988bd1754fa8f4550: Neither blocks nor log segments found. Creating new log.
I20260925 04:44:49.851572 8429 tablet_bootstrap.cc:492] T d896951b99bf46259fc1f05310408b13 P 7d5cb59e32a844c988bd1754fa8f4550: No bootstrap required, opened a new log
I20260925 04:44:49.851662 8429 ts_tablet_manager.cc:1397] T d896951b99bf46259fc1f05310408b13 P 7d5cb59e32a844c988bd1754fa8f4550: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 04:44:49.852128 8429 raft_consensus.cc:348] T d896951b99bf46259fc1f05310408b13 P 7d5cb59e32a844c988bd1754fa8f4550 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7532f3a035704530bff6b9605888f7c7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33301 } } peers { permanent_uuid: "b4886a90d7904435ada30398c846f69e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39747 } } peers { permanent_uuid: "7d5cb59e32a844c988bd1754fa8f4550" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 41117 } }
I20260925 04:44:49.852262 8429 raft_consensus.cc:374] T d896951b99bf46259fc1f05310408b13 P 7d5cb59e32a844c988bd1754fa8f4550 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:44:49.852298 8429 raft_consensus.cc:729] T d896951b99bf46259fc1f05310408b13 P 7d5cb59e32a844c988bd1754fa8f4550 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d5cb59e32a844c988bd1754fa8f4550, State: Initialized, Role: FOLLOWER
I20260925 04:44:49.852451 8429 consensus_queue.cc:260] T d896951b99bf46259fc1f05310408b13 P 7d5cb59e32a844c988bd1754fa8f4550 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7532f3a035704530bff6b9605888f7c7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33301 } } peers { permanent_uuid: "b4886a90d7904435ada30398c846f69e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39747 } } peers { permanent_uuid: "7d5cb59e32a844c988bd1754fa8f4550" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 41117 } }
I20260925 04:44:49.852721 8429 ts_tablet_manager.cc:1428] T d896951b99bf46259fc1f05310408b13 P 7d5cb59e32a844c988bd1754fa8f4550: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 04:44:49.865965 8437 raft_consensus.cc:482] T d896951b99bf46259fc1f05310408b13 P b4886a90d7904435ada30398c846f69e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:44:49.866110 8437 raft_consensus.cc:504] T d896951b99bf46259fc1f05310408b13 P b4886a90d7904435ada30398c846f69e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7532f3a035704530bff6b9605888f7c7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33301 } } peers { permanent_uuid: "b4886a90d7904435ada30398c846f69e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39747 } } peers { permanent_uuid: "7d5cb59e32a844c988bd1754fa8f4550" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 41117 } }
I20260925 04:44:49.867257 8437 leader_election.cc:288] T d896951b99bf46259fc1f05310408b13 P b4886a90d7904435ada30398c846f69e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7532f3a035704530bff6b9605888f7c7 (127.0.98.65:33301), 7d5cb59e32a844c988bd1754fa8f4550 (127.0.98.67:41117)
I20260925 04:44:49.873273 8513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:49.867530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:41117 (local address 127.0.98.66:35487)
0925 04:44:49.867921 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:49.867933 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:49.868073 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:49.868741 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:49.868749 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:49.868769 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:44:49.868899 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:49.868908 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:49.870059 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:49.870062 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:49.871600 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:49.871617 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:49.872737 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:49.872741 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:49.872841 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:49.872865 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 04:44:49.872973 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 04:44:49.873060 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"mutex_wait_us":29,"thread_start_us":106,"threads_started":1}
I20260925 04:44:49.873314 8496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:49.867637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:35487 (local address 127.0.98.67:41117)
0925 04:44:49.867771 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:49.867778 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:49.868059 (+ 281us) server_negotiation.cc:407] Connection header received
0925 04:44:49.868240 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:49.868243 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:49.868292 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:49.868367 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:44:49.869023 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:49.869945 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:49.871767 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:49.872445 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:49.872537 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:49.872605 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:49.873043 (+ 438us) server_negotiation.cc:299] Negotiation successful
0925 04:44:49.873184 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 04:44:49.873771 8512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:49.867530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:33301 (local address 127.0.98.66:53735)
0925 04:44:49.867872 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:49.867890 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:49.868021 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:49.868446 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:49.868451 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:49.868476 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:44:49.868626 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:49.868636 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:49.870042 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:49.870050 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:49.872111 (+ 2061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:49.872122 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:49.873302 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:49.873306 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:49.873408 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:49.873439 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 04:44:49.873506 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 04:44:49.873607 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":32,"thread_start_us":75,"threads_started":1}
I20260925 04:44:49.873967 8355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d896951b99bf46259fc1f05310408b13" candidate_uuid: "b4886a90d7904435ada30398c846f69e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d5cb59e32a844c988bd1754fa8f4550" is_pre_election: true
I20260925 04:44:49.874130 8514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:49.867637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:53735 (local address 127.0.98.65:33301)
0925 04:44:49.867873 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:49.867879 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:49.867965 (+ 86us) server_negotiation.cc:407] Connection header received
0925 04:44:49.868179 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:49.868199 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:49.868275 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:49.868374 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:44:49.868750 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:49.869711 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:49.872250 (+ 2539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:49.873085 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:49.873181 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:49.873239 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:49.873915 (+ 676us) server_negotiation.cc:299] Negotiation successful
0925 04:44:49.873998 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":74,"threads_started":1}
I20260925 04:44:49.874228 8355 raft_consensus.cc:2454] T d896951b99bf46259fc1f05310408b13 P 7d5cb59e32a844c988bd1754fa8f4550 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4886a90d7904435ada30398c846f69e in term 0.
I20260925 04:44:49.874771 8147 leader_election.cc:302] T d896951b99bf46259fc1f05310408b13 P b4886a90d7904435ada30398c846f69e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d5cb59e32a844c988bd1754fa8f4550, b4886a90d7904435ada30398c846f69e; no voters:
I20260925 04:44:49.874949 8075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d896951b99bf46259fc1f05310408b13" candidate_uuid: "b4886a90d7904435ada30398c846f69e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7532f3a035704530bff6b9605888f7c7" is_pre_election: true
I20260925 04:44:49.875147 8075 raft_consensus.cc:2454] T d896951b99bf46259fc1f05310408b13 P 7532f3a035704530bff6b9605888f7c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4886a90d7904435ada30398c846f69e in term 0.
I20260925 04:44:49.875758 8437 raft_consensus.cc:2781] T d896951b99bf46259fc1f05310408b13 P b4886a90d7904435ada30398c846f69e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:44:49.875819 8437 raft_consensus.cc:482] T d896951b99bf46259fc1f05310408b13 P b4886a90d7904435ada30398c846f69e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:44:49.875847 8437 raft_consensus.cc:3037] T d896951b99bf46259fc1f05310408b13 P b4886a90d7904435ada30398c846f69e [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:44:49.877094 8437 raft_consensus.cc:504] T d896951b99bf46259fc1f05310408b13 P b4886a90d7904435ada30398c846f69e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7532f3a035704530bff6b9605888f7c7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33301 } } peers { permanent_uuid: "b4886a90d7904435ada30398c846f69e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39747 } } peers { permanent_uuid: "7d5cb59e32a844c988bd1754fa8f4550" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 41117 } }
I20260925 04:44:49.877508 8437 leader_election.cc:288] T d896951b99bf46259fc1f05310408b13 P b4886a90d7904435ada30398c846f69e [CANDIDATE]: Term 1 election: Requested vote from peers 7532f3a035704530bff6b9605888f7c7 (127.0.98.65:33301), 7d5cb59e32a844c988bd1754fa8f4550 (127.0.98.67:41117)
I20260925 04:44:49.877862 8075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d896951b99bf46259fc1f05310408b13" candidate_uuid: "b4886a90d7904435ada30398c846f69e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7532f3a035704530bff6b9605888f7c7"
I20260925 04:44:49.877897 8355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d896951b99bf46259fc1f05310408b13" candidate_uuid: "b4886a90d7904435ada30398c846f69e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d5cb59e32a844c988bd1754fa8f4550"
I20260925 04:44:49.877981 8075 raft_consensus.cc:3037] T d896951b99bf46259fc1f05310408b13 P 7532f3a035704530bff6b9605888f7c7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:44:49.878019 8355 raft_consensus.cc:3037] T d896951b99bf46259fc1f05310408b13 P 7d5cb59e32a844c988bd1754fa8f4550 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:44:49.879220 8355 raft_consensus.cc:2454] T d896951b99bf46259fc1f05310408b13 P 7d5cb59e32a844c988bd1754fa8f4550 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4886a90d7904435ada30398c846f69e in term 1.
I20260925 04:44:49.879287 8075 raft_consensus.cc:2454] T d896951b99bf46259fc1f05310408b13 P 7532f3a035704530bff6b9605888f7c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4886a90d7904435ada30398c846f69e in term 1.
I20260925 04:44:49.879558 8147 leader_election.cc:302] T d896951b99bf46259fc1f05310408b13 P b4886a90d7904435ada30398c846f69e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d5cb59e32a844c988bd1754fa8f4550, b4886a90d7904435ada30398c846f69e; no voters:
I20260925 04:44:49.879762 8437 raft_consensus.cc:2781] T d896951b99bf46259fc1f05310408b13 P b4886a90d7904435ada30398c846f69e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:44:49.880007 8437 raft_consensus.cc:686] T d896951b99bf46259fc1f05310408b13 P b4886a90d7904435ada30398c846f69e [term 1 LEADER]: Becoming Leader. State: Replica: b4886a90d7904435ada30398c846f69e, State: Running, Role: LEADER
I20260925 04:44:49.880257 8437 consensus_queue.cc:237] T d896951b99bf46259fc1f05310408b13 P b4886a90d7904435ada30398c846f69e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7532f3a035704530bff6b9605888f7c7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33301 } } peers { permanent_uuid: "b4886a90d7904435ada30398c846f69e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39747 } } peers { permanent_uuid: "7d5cb59e32a844c988bd1754fa8f4550" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 41117 } }
I20260925 04:44:49.882634 7916 catalog_manager.cc:5495] T d896951b99bf46259fc1f05310408b13 P b4886a90d7904435ada30398c846f69e reported cstate change: term changed from 0 to 1, leader changed from <none> to b4886a90d7904435ada30398c846f69e (127.0.98.66). New cstate: current_term: 1 leader_uuid: "b4886a90d7904435ada30398c846f69e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7532f3a035704530bff6b9605888f7c7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4886a90d7904435ada30398c846f69e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d5cb59e32a844c988bd1754fa8f4550" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 41117 } health_report { overall_health: UNKNOWN } } }
I20260925 04:44:49.900910 8517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:49.896712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60684 (local address 127.0.98.66:39747)
0925 04:44:49.897006 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:49.897013 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:49.897024 (+ 11us) server_negotiation.cc:407] Connection header received
0925 04:44:49.897102 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:49.897104 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:49.897158 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:49.897290 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:44:49.897520 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:49.898088 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:49.899120 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:49.899450 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:49.899601 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:49.899912 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:44:49.900448 (+ 536us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:44:49.900582 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:49.900660 (+ 78us) server_negotiation.cc:299] Negotiation successful
0925 04:44:49.900721 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":120,"threads_started":1}
I20260925 04:44:49.912607 8496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:49.908895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:36399 (local address 127.0.98.67:41117)
0925 04:44:49.909059 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:49.909064 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:49.909247 (+ 183us) server_negotiation.cc:407] Connection header received
0925 04:44:49.909382 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:49.909391 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:49.909446 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:49.909659 (+ 213us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:44:49.910029 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:49.910623 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:49.911587 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:49.911803 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:49.911903 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:49.912226 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:44:49.912295 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:44:49.912384 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:49.912437 (+ 53us) server_negotiation.cc:299] Negotiation successful
0925 04:44:49.912487 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 04:44:49.912647 8520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:49.908774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:41117 (local address 127.0.98.66:36399)
0925 04:44:49.909141 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:49.909156 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:49.909286 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:49.909800 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:49.909802 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:49.909810 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 04:44:49.909909 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:49.909914 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:49.910758 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:49.910760 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:49.911481 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:49.911491 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:49.911918 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:49.911922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:49.912046 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:49.912098 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 04:44:49.912364 (+ 266us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 04:44:49.912369 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 04:44:49.912421 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 04:44:49.912486 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":113,"threads_started":1}
I20260925 04:44:49.920418 8075 raft_consensus.cc:1261] T d896951b99bf46259fc1f05310408b13 P 7532f3a035704530bff6b9605888f7c7 [term 1 FOLLOWER]: Refusing update from remote peer b4886a90d7904435ada30398c846f69e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:44:49.920473 8355 raft_consensus.cc:1261] T d896951b99bf46259fc1f05310408b13 P 7d5cb59e32a844c988bd1754fa8f4550 [term 1 FOLLOWER]: Refusing update from remote peer b4886a90d7904435ada30398c846f69e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:44:49.920854 8437 consensus_queue.cc:1035] T d896951b99bf46259fc1f05310408b13 P b4886a90d7904435ada30398c846f69e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7532f3a035704530bff6b9605888f7c7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:44:49.921058 8515 consensus_queue.cc:1035] T d896951b99bf46259fc1f05310408b13 P b4886a90d7904435ada30398c846f69e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d5cb59e32a844c988bd1754fa8f4550" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 41117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:44:49.939254 8517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:49.935842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:47203 (local address 127.0.98.66:39747)
0925 04:44:49.935953 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:49.935958 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:49.936202 (+ 244us) server_negotiation.cc:407] Connection header received
0925 04:44:49.936314 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:49.936317 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:49.936364 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:49.936430 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:44:49.936597 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:49.937110 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:49.938067 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:49.938426 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:49.938540 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:49.938793 (+ 253us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:44:49.938881 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:44:49.938951 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:49.939059 (+ 108us) server_negotiation.cc:299] Negotiation successful
0925 04:44:49.939110 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 04:44:49.939256 8525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:49.935755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:39747 (local address 127.0.98.67:47203)
0925 04:44:49.936112 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:49.936127 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:49.936225 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:49.936437 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:49.936438 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:49.936444 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 04:44:49.936534 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:49.936539 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:49.937200 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:49.937202 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:49.937959 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:49.937970 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:49.938558 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:49.938561 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:49.938632 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:49.938659 (+ 27us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 04:44:49.938978 (+ 319us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 04:44:49.938985 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 04:44:49.939047 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 04:44:49.939128 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":111,"threads_started":1}
I20260925 04:44:49.961793 7918 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51052:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:44:49.961946 7918 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51052:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:44:49.964355 7918 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1f45ce55bb3f415c8d7178965484a569: Sending DeleteTablet for 3 replicas of tablet d896951b99bf46259fc1f05310408b13
I20260925 04:44:49.965091 8195 tablet_service.cc:1514] Processing DeleteTablet for tablet d896951b99bf46259fc1f05310408b13 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:44:49 UTC) from {username='oryx', principal='oryx/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:34876
I20260925 04:44:49.965209 8334 tablet_service.cc:1514] Processing DeleteTablet for tablet d896951b99bf46259fc1f05310408b13 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:44:49 UTC) from {username='oryx', principal='oryx/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:36804
I20260925 04:44:49.965205 8055 tablet_service.cc:1514] Processing DeleteTablet for tablet d896951b99bf46259fc1f05310408b13 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:44:49 UTC) from {username='oryx', principal='oryx/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:41128
I20260925 04:44:49.965303 8507 tablet_replica.cc:330] T d896951b99bf46259fc1f05310408b13 P b4886a90d7904435ada30398c846f69e: stopping tablet replica
I20260925 04:44:49.965372 8508 tablet_replica.cc:330] T d896951b99bf46259fc1f05310408b13 P 7d5cb59e32a844c988bd1754fa8f4550: stopping tablet replica
I20260925 04:44:49.965420 8507 raft_consensus.cc:2229] T d896951b99bf46259fc1f05310408b13 P b4886a90d7904435ada30398c846f69e [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:44:49.965484 8508 raft_consensus.cc:2229] T d896951b99bf46259fc1f05310408b13 P 7d5cb59e32a844c988bd1754fa8f4550 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:44:49.965610 8508 raft_consensus.cc:2258] T d896951b99bf46259fc1f05310408b13 P 7d5cb59e32a844c988bd1754fa8f4550 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:44:49.965735 8507 raft_consensus.cc:2258] T d896951b99bf46259fc1f05310408b13 P b4886a90d7904435ada30398c846f69e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:44:49.965901 8506 tablet_replica.cc:330] T d896951b99bf46259fc1f05310408b13 P 7532f3a035704530bff6b9605888f7c7: stopping tablet replica
I20260925 04:44:49.966013 8506 raft_consensus.cc:2229] T d896951b99bf46259fc1f05310408b13 P 7532f3a035704530bff6b9605888f7c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:44:49.966127 8506 raft_consensus.cc:2258] T d896951b99bf46259fc1f05310408b13 P 7532f3a035704530bff6b9605888f7c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:44:49.966404 8508 ts_tablet_manager.cc:1905] T d896951b99bf46259fc1f05310408b13 P 7d5cb59e32a844c988bd1754fa8f4550: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:44:49.966423 8507 ts_tablet_manager.cc:1905] T d896951b99bf46259fc1f05310408b13 P b4886a90d7904435ada30398c846f69e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:44:49.966977 8506 ts_tablet_manager.cc:1905] T d896951b99bf46259fc1f05310408b13 P 7532f3a035704530bff6b9605888f7c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:44:49.969004 8507 ts_tablet_manager.cc:1918] T d896951b99bf46259fc1f05310408b13 P b4886a90d7904435ada30398c846f69e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 04:44:49.969033 8508 ts_tablet_manager.cc:1918] T d896951b99bf46259fc1f05310408b13 P 7d5cb59e32a844c988bd1754fa8f4550: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 04:44:49.969090 8507 log.cc:1196] T d896951b99bf46259fc1f05310408b13 P b4886a90d7904435ada30398c846f69e: Deleting WAL directory at /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/minicluster-data/ts-1/wal/wals/d896951b99bf46259fc1f05310408b13
I20260925 04:44:49.969096 8508 log.cc:1196] T d896951b99bf46259fc1f05310408b13 P 7d5cb59e32a844c988bd1754fa8f4550: Deleting WAL directory at /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/minicluster-data/ts-2/wal/wals/d896951b99bf46259fc1f05310408b13
I20260925 04:44:49.969360 8507 ts_tablet_manager.cc:1939] T d896951b99bf46259fc1f05310408b13 P b4886a90d7904435ada30398c846f69e: Deleting consensus metadata
I20260925 04:44:49.969364 8508 ts_tablet_manager.cc:1939] T d896951b99bf46259fc1f05310408b13 P 7d5cb59e32a844c988bd1754fa8f4550: Deleting consensus metadata
I20260925 04:44:49.969486 8506 ts_tablet_manager.cc:1918] T d896951b99bf46259fc1f05310408b13 P 7532f3a035704530bff6b9605888f7c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 04:44:49.969570 8506 log.cc:1196] T d896951b99bf46259fc1f05310408b13 P 7532f3a035704530bff6b9605888f7c7: Deleting WAL directory at /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790311449873301-393-0/minicluster-data/ts-0/wal/wals/d896951b99bf46259fc1f05310408b13
I20260925 04:44:49.969818 8506 ts_tablet_manager.cc:1939] T d896951b99bf46259fc1f05310408b13 P 7532f3a035704530bff6b9605888f7c7: Deleting consensus metadata
I20260925 04:44:49.970062 7903 catalog_manager.cc:4841] TS 7d5cb59e32a844c988bd1754fa8f4550 (127.0.98.67:41117): tablet d896951b99bf46259fc1f05310408b13 (table test-table [id=8bcf615b5718407c8a6a71fc4fa2d592]) successfully deleted
I20260925 04:44:49.970129 7902 catalog_manager.cc:4841] TS b4886a90d7904435ada30398c846f69e (127.0.98.66:39747): tablet d896951b99bf46259fc1f05310408b13 (table test-table [id=8bcf615b5718407c8a6a71fc4fa2d592]) successfully deleted
I20260925 04:44:49.970443 7905 catalog_manager.cc:4841] TS 7532f3a035704530bff6b9605888f7c7 (127.0.98.65:33301): tablet d896951b99bf46259fc1f05310408b13 (table test-table [id=8bcf615b5718407c8a6a71fc4fa2d592]) successfully deleted
I20260925 04:44:49.971244 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 7990
I20260925 04:44:49.980086 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 8130
I20260925 04:44:49.986522 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 8270
I20260925 04:44:49.993366 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 7886
2026-09-25T04:44:49Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3376 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790311449873301-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:44:50 dist-test-slave-kghz krb5kdc[8530](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:44:50 dist-test-slave-kghz krb5kdc[8530](info): set up 2 sockets
Sep 25 04:44:50 dist-test-slave-kghz krb5kdc[8530](info): commencing operation
krb5kdc: starting...
W20260925 04:44:51.083026 393 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790311449873301-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790311449873301-393-0/krb5kdc/test-user.keytab.
Sep 25 04:44:51 dist-test-slave-kghz krb5kdc[8530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311491, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:44:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:44:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260925 04:44:51.208912 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790311449873301-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790311449873301-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790311449873301-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:32983
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:46691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:32983
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:44:51.322960 8543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:44:51.323271 8543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:44:51.323318 8543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:44:51.327550 8543 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:44:51.327621 8543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:44:51.327641 8543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:44:51.327659 8543 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:44:51.327675 8543 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:44:51.333273 8543 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:46691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790311449873301-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790311449873301-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:32983
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:32983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8543
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790311449873301-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:44:51.334558 8543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:44:51.335628 8543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:44:51.344424 8548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:51.344425 8551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:51.344733 8543 server_base.cc:1034] running on GCE node
W20260925 04:44:51.344424 8549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:51.345372 8543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:44:51.346750 8543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:44:51.347934 8543 hybrid_clock.cc:648] HybridClock initialized: now 1790311491347901 us; error 48 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-25T04:44:51Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1362 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:44:51 dist-test-slave-kghz krb5kdc[8556](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:44:51 dist-test-slave-kghz krb5kdc[8556](info): set up 2 sockets
Sep 25 04:44:51 dist-test-slave-kghz krb5kdc[8556](info): commencing operation
krb5kdc: starting...
W20260925 04:44:52.403214 393 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/krb5kdc/test-user.keytab.
Sep 25 04:44:52 dist-test-slave-kghz krb5kdc[8556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311492, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:44:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:44:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260925 04:44:52.529529 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:34911
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:44821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:34911
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:44:52.643396 8569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:44:52.643760 8569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:44:52.643832 8569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:44:52.647953 8569 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:44:52.648097 8569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:44:52.648154 8569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:44:52.648188 8569 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:44:52.648221 8569 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:44:52.653620 8569 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:44821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:34911
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:34911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8569
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:44:52.654785 8569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:44:52.655831 8569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:44:52.666487 8575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:52.666910 8569 server_base.cc:1034] running on GCE node
W20260925 04:44:52.666509 8577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:52.666793 8574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:52.667899 8569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:44:52.669462 8569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:44:52.670672 8569 hybrid_clock.cc:648] HybridClock initialized: now 1790311492670608 us; error 57 us; skew 500 ppm
Sep 25 04:44:52 dist-test-slave-kghz krb5kdc[8556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311492, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:44:52.674952 8569 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260925 04:44:52.676225 8569 webserver.cc:460] Webserver started at http://127.0.98.126:42107/ using document root <none> and password file <none>
I20260925 04:44:52.677047 8569 fs_manager.cc:359] Metadata directory not provided
I20260925 04:44:52.677129 8569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:44:52.677373 8569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:44:52.680320 8569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/master-0/data/instance:
uuid: "b9ddd268e8b848aeaa70584a3e0391a4"
format_stamp: "Formatted at 2026-09-25 04:44:52 on dist-test-slave-kghz"
server_key: "4a8ce33259e98db553d6a533b80141c4"
server_key_iv: "0a2e39f909c8f705beb930de222d5bf1"
server_key_version: "encryptionkey@0"
I20260925 04:44:52.681162 8569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/master-0/wal/instance:
uuid: "b9ddd268e8b848aeaa70584a3e0391a4"
format_stamp: "Formatted at 2026-09-25 04:44:52 on dist-test-slave-kghz"
server_key: "4a8ce33259e98db553d6a533b80141c4"
server_key_iv: "0a2e39f909c8f705beb930de222d5bf1"
server_key_version: "encryptionkey@0"
I20260925 04:44:52.686167 8569 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 04:44:52.689435 8584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:52.691345 8569 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:44:52.691497 8569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/master-0/wal
uuid: "b9ddd268e8b848aeaa70584a3e0391a4"
format_stamp: "Formatted at 2026-09-25 04:44:52 on dist-test-slave-kghz"
server_key: "4a8ce33259e98db553d6a533b80141c4"
server_key_iv: "0a2e39f909c8f705beb930de222d5bf1"
server_key_version: "encryptionkey@0"
I20260925 04:44:52.691646 8569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:44:52.718739 8569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:44:52.723024 8569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:44:52.723237 8569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:44:52.730088 8569 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:34911
I20260925 04:44:52.730137 8636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:34911 every 8 connection(s)
I20260925 04:44:52.731151 8569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
I20260925 04:44:52.733943 8637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:44:52.736685 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 8569
I20260925 04:44:52.736858 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/master-0/wal/instance
I20260925 04:44:52.737190 393 external_mini_cluster.cc:1428] Setting key 60a6c91873c3a79f79fc8f19922b6bee
I20260925 04:44:52.740552 8637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9ddd268e8b848aeaa70584a3e0391a4: Bootstrap starting.
I20260925 04:44:52.743539 8637 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9ddd268e8b848aeaa70584a3e0391a4: Neither blocks nor log segments found. Creating new log.
I20260925 04:44:52.744364 8637 log.cc:824] T 00000000000000000000000000000000 P b9ddd268e8b848aeaa70584a3e0391a4: Log is configured to *not* fsync() on all Append() calls
I20260925 04:44:52.746703 8637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9ddd268e8b848aeaa70584a3e0391a4: No bootstrap required, opened a new log
Sep 25 04:44:52 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311492, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:44:52.750605 8637 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9ddd268e8b848aeaa70584a3e0391a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ddd268e8b848aeaa70584a3e0391a4" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34911 } }
I20260925 04:44:52.750928 8637 raft_consensus.cc:374] T 00000000000000000000000000000000 P b9ddd268e8b848aeaa70584a3e0391a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:44:52.751012 8637 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9ddd268e8b848aeaa70584a3e0391a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9ddd268e8b848aeaa70584a3e0391a4, State: Initialized, Role: FOLLOWER
I20260925 04:44:52.751503 8637 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9ddd268e8b848aeaa70584a3e0391a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ddd268e8b848aeaa70584a3e0391a4" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34911 } }
I20260925 04:44:52.751698 8637 raft_consensus.cc:388] T 00000000000000000000000000000000 P b9ddd268e8b848aeaa70584a3e0391a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:44:52.751768 8637 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9ddd268e8b848aeaa70584a3e0391a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:44:52.751865 8637 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9ddd268e8b848aeaa70584a3e0391a4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:44:52.753037 8637 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9ddd268e8b848aeaa70584a3e0391a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ddd268e8b848aeaa70584a3e0391a4" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34911 } }
I20260925 04:44:52.753391 8637 leader_election.cc:302] T 00000000000000000000000000000000 P b9ddd268e8b848aeaa70584a3e0391a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9ddd268e8b848aeaa70584a3e0391a4; no voters:
I20260925 04:44:52.753720 8637 leader_election.cc:288] T 00000000000000000000000000000000 P b9ddd268e8b848aeaa70584a3e0391a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:44:52.753808 8640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:52.738752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44840 (local address 127.0.98.126:34911)
0925 04:44:52.739269 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:52.739292 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:52.739338 (+ 46us) server_negotiation.cc:407] Connection header received
0925 04:44:52.740157 (+ 819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:52.740195 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:52.740632 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:52.741056 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:52.741734 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:52.743276 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:52.744597 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:52.745212 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:52.745387 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:52.748187 (+ 2800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:52.748192 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:52.748207 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:52.748234 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:52.750496 (+ 2262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:52.751137 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:52.751143 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:52.751154 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:52.751298 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:52.751599 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:52.751602 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:52.751604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:52.752151 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:52.752308 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:52.752603 (+ 295us) server_negotiation.cc:299] Negotiation successful
0925 04:44:52.752910 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":156,"threads_started":1}
I20260925 04:44:52.753823 8642 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9ddd268e8b848aeaa70584a3e0391a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:44:52.754010 8642 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9ddd268e8b848aeaa70584a3e0391a4 [term 1 LEADER]: Becoming Leader. State: Replica: b9ddd268e8b848aeaa70584a3e0391a4, State: Running, Role: LEADER
I20260925 04:44:52.754374 8642 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9ddd268e8b848aeaa70584a3e0391a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ddd268e8b848aeaa70584a3e0391a4" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34911 } }
I20260925 04:44:52.755072 8637 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9ddd268e8b848aeaa70584a3e0391a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:44:52.756013 8644 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9ddd268e8b848aeaa70584a3e0391a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b9ddd268e8b848aeaa70584a3e0391a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ddd268e8b848aeaa70584a3e0391a4" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34911 } } }
I20260925 04:44:52.756157 8644 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9ddd268e8b848aeaa70584a3e0391a4 [sys.catalog]: This master's current role is: LEADER
I20260925 04:44:52.756533 8643 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9ddd268e8b848aeaa70584a3e0391a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9ddd268e8b848aeaa70584a3e0391a4. Latest consensus state: current_term: 1 leader_uuid: "b9ddd268e8b848aeaa70584a3e0391a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ddd268e8b848aeaa70584a3e0391a4" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34911 } } }
I20260925 04:44:52.756641 8643 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9ddd268e8b848aeaa70584a3e0391a4 [sys.catalog]: This master's current role is: LEADER
I20260925 04:44:52.756912 8651 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:44:52.759770 8651 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:44:52.765836 8651 catalog_manager.cc:1347] Generated new cluster ID: f15d3a63d6434470a3767cc9ec27fcca
I20260925 04:44:52.765918 8651 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:44:52.791329 8651 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:44:52.792682 8651 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:44:52.806305 8651 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b9ddd268e8b848aeaa70584a3e0391a4: Generated new TSK 0
I20260925 04:44:52.807111 8651 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/krb5kdc/kudu_127.0.98.65.keytab.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/krb5kdc/kudu_127.0.98.65.keytab.
I20260925 04:44:52.845676 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:0
--local_ip_for_outbound_sockets=127.0.98.65
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:34911
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:44821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:44:52.979964 8663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:44:52.980283 8663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:44:52.980329 8663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:44:52.984436 8663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:44:52.984508 8663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:44:52.984570 8663 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260925 04:44:52.990586 8663 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:44821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:34911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8663
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:44:52.991722 8663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:44:52.992733 8663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:44:53.002923 8668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:53.002985 8671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:53.002965 8663 server_base.cc:1034] running on GCE node
W20260925 04:44:53.003139 8669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:53.003572 8663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:44:53.004220 8663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:44:53.005388 8663 hybrid_clock.cc:648] HybridClock initialized: now 1790311493005350 us; error 49 us; skew 500 ppm
Sep 25 04:44:53 dist-test-slave-kghz krb5kdc[8556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311493, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:44:53.008347 8663 init.cc:377] Logged in from keytab as kudu/127.0.98.65@KRBTEST.COM (short username kudu)
I20260925 04:44:53.009274 8663 webserver.cc:460] Webserver started at http://127.0.98.65:38575/ using document root <none> and password file <none>
I20260925 04:44:53.009971 8663 fs_manager.cc:359] Metadata directory not provided
I20260925 04:44:53.010072 8663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:44:53.010351 8663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:44:53.012459 8663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-0/data/instance:
uuid: "74ff9ff0a144486c94e1554674c80577"
format_stamp: "Formatted at 2026-09-25 04:44:53 on dist-test-slave-kghz"
server_key: "ef382a7dcb7f4ac8cede597f294ceae8"
server_key_iv: "51e0d91831ee940f79b62e2235163d7e"
server_key_version: "encryptionkey@0"
I20260925 04:44:53.013056 8663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-0/wal/instance:
uuid: "74ff9ff0a144486c94e1554674c80577"
format_stamp: "Formatted at 2026-09-25 04:44:53 on dist-test-slave-kghz"
server_key: "ef382a7dcb7f4ac8cede597f294ceae8"
server_key_iv: "51e0d91831ee940f79b62e2235163d7e"
server_key_version: "encryptionkey@0"
I20260925 04:44:53.017143 8663 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 04:44:53.019747 8678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:53.020998 8663 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 04:44:53.021176 8663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-0/wal
uuid: "74ff9ff0a144486c94e1554674c80577"
format_stamp: "Formatted at 2026-09-25 04:44:53 on dist-test-slave-kghz"
server_key: "ef382a7dcb7f4ac8cede597f294ceae8"
server_key_iv: "51e0d91831ee940f79b62e2235163d7e"
server_key_version: "encryptionkey@0"
I20260925 04:44:53.021289 8663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:44:53.038945 8663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:44:53.042800 8663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:44:53.043023 8663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:44:53.043635 8663 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:44:53.044476 8663 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:44:53.044545 8663 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:53.044610 8663 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:44:53.044655 8663 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:53.053364 8663 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:34263
I20260925 04:44:53.053383 8791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:34263 every 8 connection(s)
I20260925 04:44:53.054378 8663 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
I20260925 04:44:53.063242 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 8663
I20260925 04:44:53.063495 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-0/wal/instance
I20260925 04:44:53.063874 393 external_mini_cluster.cc:1428] Setting key c5120057e15560e2e4f473550366c0c2
Sep 25 04:44:53 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311493, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:44:53.070326 8640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:53.056361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:39781 (local address 127.0.98.126:34911)
0925 04:44:53.056506 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:53.056511 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:53.057632 (+ 1121us) server_negotiation.cc:407] Connection header received
0925 04:44:53.059202 (+ 1570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:53.059205 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:53.059254 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:53.059338 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:53.060655 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:53.061359 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:53.062675 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:53.062909 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:53.063061 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:53.066454 (+ 3393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:53.066458 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:53.066460 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:53.066486 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:53.068112 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:53.068683 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:53.068687 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:53.068688 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:53.068765 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:53.069110 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:53.069114 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:53.069116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:53.069307 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:53.069410 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:53.069947 (+ 537us) server_negotiation.cc:299] Negotiation successful
0925 04:44:53.070092 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 04:44:53.071200 8794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:53.056954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34911 (local address 127.0.98.65:39781)
0925 04:44:53.057455 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:53.057506 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:53.058975 (+ 1469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:53.059550 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:53.059557 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:53.059935 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:53.060480 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:53.060495 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:53.061519 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:53.061522 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:53.062529 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:53.062539 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:53.063250 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:53.063253 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:53.063415 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:53.064225 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:53.064244 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:53.066265 (+ 2021us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:44:53.068248 (+ 1983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:53.068257 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:53.068277 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:53.068569 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:53.068862 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:53.068865 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:53.068867 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:53.069016 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:53.069410 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:44:53.069419 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:44:53.069700 (+ 281us) client_negotiation.cc:769] Sending connection context
0925 04:44:53.069888 (+ 188us) client_negotiation.cc:240] Negotiation successful
0925 04:44:53.070315 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":117,"threads_started":1}
I20260925 04:44:53.072536 8792 heartbeater.cc:347] Connected to a master server at 127.0.98.126:34911
I20260925 04:44:53.072826 8792 heartbeater.cc:464] Registering TS with master...
I20260925 04:44:53.073500 8792 heartbeater.cc:511] Master 127.0.98.126:34911 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.66@KRBTEST.COM; defaulting to no policy
I20260925 04:44:53.075243 8601 ts_manager.cc:194] Registered new tserver with Master: 74ff9ff0a144486c94e1554674c80577 (127.0.98.65:34263)
I20260925 04:44:53.076505 8601 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.65@KRBTEST.COM'} at 127.0.98.65:39781
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/krb5kdc/kudu_127.0.98.66.keytab.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/krb5kdc/kudu_127.0.98.66.keytab.
I20260925 04:44:53.093348 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:0
--local_ip_for_outbound_sockets=127.0.98.66
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:34911
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:44821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:44:53.207646 8797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:44:53.207934 8797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:44:53.208026 8797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:44:53.212409 8797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:44:53.212474 8797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:44:53.212574 8797 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260925 04:44:53.218631 8797 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:44821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:34911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:44:53.219758 8797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:44:53.220798 8797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:44:53.231205 8803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:53.231244 8805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:53.231222 8802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:53.231544 8797 server_base.cc:1034] running on GCE node
I20260925 04:44:53.231997 8797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:44:53.232718 8797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:44:53.233886 8797 hybrid_clock.cc:648] HybridClock initialized: now 1790311493233828 us; error 77 us; skew 500 ppm
Sep 25 04:44:53 dist-test-slave-kghz krb5kdc[8556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311493, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:44:53.236836 8797 init.cc:377] Logged in from keytab as kudu/127.0.98.66@KRBTEST.COM (short username kudu)
I20260925 04:44:53.237743 8797 webserver.cc:460] Webserver started at http://127.0.98.66:40779/ using document root <none> and password file <none>
I20260925 04:44:53.238377 8797 fs_manager.cc:359] Metadata directory not provided
I20260925 04:44:53.238469 8797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:44:53.238704 8797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:44:53.240877 8797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-1/data/instance:
uuid: "bf746411a6fa48b7b94010f9d7783bbd"
format_stamp: "Formatted at 2026-09-25 04:44:53 on dist-test-slave-kghz"
server_key: "87bf5bdba37a7a4c21772523428f2bb2"
server_key_iv: "9347ef49c018ab28543bdc4170fa51cd"
server_key_version: "encryptionkey@0"
I20260925 04:44:53.241446 8797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-1/wal/instance:
uuid: "bf746411a6fa48b7b94010f9d7783bbd"
format_stamp: "Formatted at 2026-09-25 04:44:53 on dist-test-slave-kghz"
server_key: "87bf5bdba37a7a4c21772523428f2bb2"
server_key_iv: "9347ef49c018ab28543bdc4170fa51cd"
server_key_version: "encryptionkey@0"
I20260925 04:44:53.245445 8797 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 04:44:53.248106 8812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:53.249469 8797 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 04:44:53.249606 8797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-1/wal
uuid: "bf746411a6fa48b7b94010f9d7783bbd"
format_stamp: "Formatted at 2026-09-25 04:44:53 on dist-test-slave-kghz"
server_key: "87bf5bdba37a7a4c21772523428f2bb2"
server_key_iv: "9347ef49c018ab28543bdc4170fa51cd"
server_key_version: "encryptionkey@0"
I20260925 04:44:53.249727 8797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:44:53.262620 8797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:44:53.266026 8797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:44:53.266315 8797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:44:53.266925 8797 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:44:53.267779 8797 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:44:53.267848 8797 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:53.267917 8797 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:44:53.267963 8797 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:53.276495 8797 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:35501
I20260925 04:44:53.276554 8925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:35501 every 8 connection(s)
I20260925 04:44:53.277499 8797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
I20260925 04:44:53.280056 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 8797
I20260925 04:44:53.280179 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-1/wal/instance
I20260925 04:44:53.280493 393 external_mini_cluster.cc:1428] Setting key ad9571f1895050660b5d0f0968a50198
Sep 25 04:44:53 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311493, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.67@KRBTEST.COM" created.
I20260925 04:44:53.292460 8640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:53.279491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:48861 (local address 127.0.98.126:34911)
0925 04:44:53.279649 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:53.279653 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:53.280805 (+ 1152us) server_negotiation.cc:407] Connection header received
0925 04:44:53.282426 (+ 1621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:53.282429 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:53.282483 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:53.282585 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:53.283802 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:53.284331 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:53.285489 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:53.285716 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:53.285824 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:53.288436 (+ 2612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:53.288439 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:53.288441 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:53.288466 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:53.290059 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:53.290672 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:53.290678 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:53.290682 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:53.290767 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:53.291127 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:53.291133 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:53.291137 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:53.291394 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:53.291497 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:53.292139 (+ 642us) server_negotiation.cc:299] Negotiation successful
0925 04:44:53.292282 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 04:44:53.293319 8928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:53.279990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34911 (local address 127.0.98.66:48861)
0925 04:44:53.280651 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:53.280696 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:53.282166 (+ 1470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:53.282781 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:53.282789 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:53.283153 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:53.283658 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:53.283672 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:53.284457 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:53.284461 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:53.285366 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:53.285374 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:53.285863 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:53.285867 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:53.286041 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:53.286745 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:53.286762 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:53.288257 (+ 1495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:44:53.290192 (+ 1935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:53.290201 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:53.290220 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:53.290560 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:53.290869 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:53.290872 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:53.290874 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:53.291025 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:53.291507 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:44:53.291516 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:44:53.291834 (+ 318us) client_negotiation.cc:769] Sending connection context
0925 04:44:53.292084 (+ 250us) client_negotiation.cc:240] Negotiation successful
0925 04:44:53.292433 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":168,"threads_started":1}
I20260925 04:44:53.294756 8926 heartbeater.cc:347] Connected to a master server at 127.0.98.126:34911
I20260925 04:44:53.295011 8926 heartbeater.cc:464] Registering TS with master...
I20260925 04:44:53.295549 8926 heartbeater.cc:511] Master 127.0.98.126:34911 requested a full tablet report, sending...
I20260925 04:44:53.296934 8601 ts_manager.cc:194] Registered new tserver with Master: bf746411a6fa48b7b94010f9d7783bbd (127.0.98.66:35501)
I20260925 04:44:53.297884 8601 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.66@KRBTEST.COM'} at 127.0.98.66:48861
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/krb5kdc/kudu_127.0.98.67.keytab.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/krb5kdc/kudu_127.0.98.67.keytab.
I20260925 04:44:53.307405 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:0
--local_ip_for_outbound_sockets=127.0.98.67
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:34911
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:44821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:44:53.424269 8931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:44:53.424634 8931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:44:53.424685 8931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:44:53.429421 8931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:44:53.429495 8931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:44:53.429564 8931 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260925 04:44:53.435242 8931 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:44821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:34911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:44:53.436369 8931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:44:53.437445 8931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:44:53.448032 8937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:53.447985 8936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:53.448016 8931 server_base.cc:1034] running on GCE node
W20260925 04:44:53.447945 8939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:53.448782 8931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:44:53.449493 8931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:44:53.450707 8931 hybrid_clock.cc:648] HybridClock initialized: now 1790311493450687 us; error 33 us; skew 500 ppm
Sep 25 04:44:53 dist-test-slave-kghz krb5kdc[8556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311493, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:44:53.454123 8931 init.cc:377] Logged in from keytab as kudu/127.0.98.67@KRBTEST.COM (short username kudu)
I20260925 04:44:53.455205 8931 webserver.cc:460] Webserver started at http://127.0.98.67:43629/ using document root <none> and password file <none>
I20260925 04:44:53.455837 8931 fs_manager.cc:359] Metadata directory not provided
I20260925 04:44:53.455902 8931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:44:53.456144 8931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:44:53.458408 8931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-2/data/instance:
uuid: "35dc94a4ad3541c99f3f738f0f0a5ef0"
format_stamp: "Formatted at 2026-09-25 04:44:53 on dist-test-slave-kghz"
server_key: "b7b783587a754635868474a5861e0e8c"
server_key_iv: "24a570e0d590d745215b630bc5f97ab6"
server_key_version: "encryptionkey@0"
I20260925 04:44:53.459010 8931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-2/wal/instance:
uuid: "35dc94a4ad3541c99f3f738f0f0a5ef0"
format_stamp: "Formatted at 2026-09-25 04:44:53 on dist-test-slave-kghz"
server_key: "b7b783587a754635868474a5861e0e8c"
server_key_iv: "24a570e0d590d745215b630bc5f97ab6"
server_key_version: "encryptionkey@0"
I20260925 04:44:53.463057 8931 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 04:44:53.465798 8946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:53.467411 8931 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:44:53.467649 8931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-2/wal
uuid: "35dc94a4ad3541c99f3f738f0f0a5ef0"
format_stamp: "Formatted at 2026-09-25 04:44:53 on dist-test-slave-kghz"
server_key: "b7b783587a754635868474a5861e0e8c"
server_key_iv: "24a570e0d590d745215b630bc5f97ab6"
server_key_version: "encryptionkey@0"
I20260925 04:44:53.467777 8931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:44:53.488116 8931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:44:53.491698 8931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:44:53.491927 8931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:44:53.492585 8931 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:44:53.493507 8931 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:44:53.493557 8931 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:53.493621 8931 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:44:53.493655 8931 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:53.503047 8931 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:44447
I20260925 04:44:53.503064 9059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:44447 every 8 connection(s)
I20260925 04:44:53.503996 8931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
I20260925 04:44:53.504365 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 8931
I20260925 04:44:53.504521 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-2/wal/instance
I20260925 04:44:53.504845 393 external_mini_cluster.cc:1428] Setting key 9d9da972505f6c1facae5e8fac3424a6
Sep 25 04:44:53 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311493, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:44:53.518908 8640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:53.505874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:42031 (local address 127.0.98.126:34911)
0925 04:44:53.506012 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:53.506016 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:53.507152 (+ 1136us) server_negotiation.cc:407] Connection header received
0925 04:44:53.508594 (+ 1442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:53.508597 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:53.508642 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:53.508726 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:53.509921 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:53.510670 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:53.511834 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:53.512091 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:53.512179 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:53.514867 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:53.514870 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:53.514872 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:53.514913 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:53.516357 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:53.516976 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:53.517025 (+ 49us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:53.517026 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:53.517111 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:53.517505 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:53.517507 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:53.517509 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:53.517774 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:53.517857 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:53.518608 (+ 751us) server_negotiation.cc:299] Negotiation successful
0925 04:44:53.518749 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 04:44:53.519922 9062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:53.506474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34911 (local address 127.0.98.67:42031)
0925 04:44:53.507007 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:53.507065 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:53.508376 (+ 1311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:53.508863 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:53.508870 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:53.509268 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:53.509778 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:53.509793 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:53.510796 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:53.510799 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:53.511715 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:53.511724 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:53.512239 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:53.512241 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:53.512392 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:53.513082 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:53.513100 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:53.514644 (+ 1544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:44:53.516466 (+ 1822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:53.516472 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:53.516485 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:53.516868 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:53.517197 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:44:53.517200 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:44:53.517201 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:44:53.517396 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:44:53.517875 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:44:53.517881 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:44:53.518304 (+ 423us) client_negotiation.cc:769] Sending connection context
0925 04:44:53.518538 (+ 234us) client_negotiation.cc:240] Negotiation successful
0925 04:44:53.518901 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":143,"threads_started":1}
I20260925 04:44:53.521131 9060 heartbeater.cc:347] Connected to a master server at 127.0.98.126:34911
I20260925 04:44:53.521382 9060 heartbeater.cc:464] Registering TS with master...
I20260925 04:44:53.521893 9060 heartbeater.cc:511] Master 127.0.98.126:34911 requested a full tablet report, sending...
I20260925 04:44:53.523108 8601 ts_manager.cc:194] Registered new tserver with Master: 35dc94a4ad3541c99f3f738f0f0a5ef0 (127.0.98.67:44447)
I20260925 04:44:53.523684 8601 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.67@KRBTEST.COM'} at 127.0.98.67:42031
I20260925 04:44:53.531416 393 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/krb5kdc/oryx_127.0.98.65.keytab.
Entry for principal oryx/127.0.98.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/krb5kdc/oryx_127.0.98.65.keytab.
WARNING: no policy specified for oryx/127.0.98.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/krb5kdc/oryx_127.0.98.66.keytab.
Entry for principal oryx/127.0.98.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/krb5kdc/oryx_127.0.98.66.keytab.
WARNING: no policy specified for oryx/127.0.98.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/krb5kdc/oryx_127.0.98.67.keytab.
Entry for principal oryx/127.0.98.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/krb5kdc/oryx_127.0.98.67.keytab.
I20260925 04:44:53.610509 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 8663
I20260925 04:44:53.616372 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 8797
I20260925 04:44:53.622155 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 8931
I20260925 04:44:53.627580 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 8569
I20260925 04:44:53.634306 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:34911
--webserver_interface=127.0.98.126
--webserver_port=42107
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:44821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:34911
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:44:53.751955 9069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:44:53.752271 9069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:44:53.752317 9069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:44:53.756642 9069 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:44:53.756726 9069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:44:53.756747 9069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:44:53.756764 9069 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:44:53.756780 9069 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:44:53.762454 9069 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:44821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:34911
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:34911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=42107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9069
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:44:53.763541 9069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:44:53.764562 9069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:44:53.773483 9075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:53.773594 9077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:53.773483 9074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:53.773806 9069 server_base.cc:1034] running on GCE node
I20260925 04:44:53.774459 9069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:44:53.775713 9069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:44:53.776950 9069 hybrid_clock.cc:648] HybridClock initialized: now 1790311493776879 us; error 73 us; skew 500 ppm
Sep 25 04:44:53 dist-test-slave-kghz krb5kdc[8556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311493, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:44:53.780066 9069 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260925 04:44:53.781041 9069 webserver.cc:460] Webserver started at http://127.0.98.126:42107/ using document root <none> and password file <none>
I20260925 04:44:53.781629 9069 fs_manager.cc:359] Metadata directory not provided
I20260925 04:44:53.781723 9069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:44:53.785756 9069 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 04:44:53.788082 9084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:53.789371 9069 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 04:44:53.789503 9069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/master-0/wal
uuid: "b9ddd268e8b848aeaa70584a3e0391a4"
format_stamp: "Formatted at 2026-09-25 04:44:52 on dist-test-slave-kghz"
server_key: "4a8ce33259e98db553d6a533b80141c4"
server_key_iv: "0a2e39f909c8f705beb930de222d5bf1"
server_key_version: "encryptionkey@0"
I20260925 04:44:53.790083 9069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:44:53.799662 9069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:44:53.803000 9069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:44:53.803257 9069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:44:53.810192 9069 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:34911
I20260925 04:44:53.810292 9136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:34911 every 8 connection(s)
I20260925 04:44:53.811372 9069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
I20260925 04:44:53.815953 9137 sys_catalog.cc:263] Verifying existing consensus state
I20260925 04:44:53.817863 9137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9ddd268e8b848aeaa70584a3e0391a4: Bootstrap starting.
I20260925 04:44:53.821086 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 9069
I20260925 04:44:53.821564 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:34263
--local_ip_for_outbound_sockets=127.0.98.65
--tserver_master_addrs=127.0.98.126:34911
--webserver_port=38575
--webserver_interface=127.0.98.65
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:44821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.65
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/krb5kdc/oryx_127.0.98.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 04:44:53.835795 9137 log.cc:824] T 00000000000000000000000000000000 P b9ddd268e8b848aeaa70584a3e0391a4: Log is configured to *not* fsync() on all Append() calls
I20260925 04:44:53.842800 9137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9ddd268e8b848aeaa70584a3e0391a4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 04:44:53.843209 9137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9ddd268e8b848aeaa70584a3e0391a4: Bootstrap complete.
I20260925 04:44:53.847681 9137 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9ddd268e8b848aeaa70584a3e0391a4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ddd268e8b848aeaa70584a3e0391a4" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34911 } }
I20260925 04:44:53.848121 9137 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9ddd268e8b848aeaa70584a3e0391a4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9ddd268e8b848aeaa70584a3e0391a4, State: Initialized, Role: FOLLOWER
I20260925 04:44:53.848671 9137 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9ddd268e8b848aeaa70584a3e0391a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ddd268e8b848aeaa70584a3e0391a4" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34911 } }
I20260925 04:44:53.848820 9137 raft_consensus.cc:388] T 00000000000000000000000000000000 P b9ddd268e8b848aeaa70584a3e0391a4 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:44:53.848904 9137 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9ddd268e8b848aeaa70584a3e0391a4 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:44:53.849018 9137 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9ddd268e8b848aeaa70584a3e0391a4 [term 1 FOLLOWER]: Advancing to term 2
I20260925 04:44:53.851157 9137 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9ddd268e8b848aeaa70584a3e0391a4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ddd268e8b848aeaa70584a3e0391a4" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34911 } }
I20260925 04:44:53.851524 9137 leader_election.cc:302] T 00000000000000000000000000000000 P b9ddd268e8b848aeaa70584a3e0391a4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9ddd268e8b848aeaa70584a3e0391a4; no voters:
I20260925 04:44:53.851871 9137 leader_election.cc:288] T 00000000000000000000000000000000 P b9ddd268e8b848aeaa70584a3e0391a4 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 04:44:53.852041 9141 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9ddd268e8b848aeaa70584a3e0391a4 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 04:44:53.852324 9141 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9ddd268e8b848aeaa70584a3e0391a4 [term 2 LEADER]: Becoming Leader. State: Replica: b9ddd268e8b848aeaa70584a3e0391a4, State: Running, Role: LEADER
I20260925 04:44:53.852766 9141 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9ddd268e8b848aeaa70584a3e0391a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ddd268e8b848aeaa70584a3e0391a4" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34911 } }
I20260925 04:44:53.853163 9137 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9ddd268e8b848aeaa70584a3e0391a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:44:53.853823 9141 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9ddd268e8b848aeaa70584a3e0391a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9ddd268e8b848aeaa70584a3e0391a4. Latest consensus state: current_term: 2 leader_uuid: "b9ddd268e8b848aeaa70584a3e0391a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ddd268e8b848aeaa70584a3e0391a4" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34911 } } }
I20260925 04:44:53.854041 9141 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9ddd268e8b848aeaa70584a3e0391a4 [sys.catalog]: This master's current role is: LEADER
I20260925 04:44:53.854295 9142 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9ddd268e8b848aeaa70584a3e0391a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b9ddd268e8b848aeaa70584a3e0391a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ddd268e8b848aeaa70584a3e0391a4" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34911 } } }
I20260925 04:44:53.854463 9142 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9ddd268e8b848aeaa70584a3e0391a4 [sys.catalog]: This master's current role is: LEADER
I20260925 04:44:53.854482 9149 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:44:53.857944 9149 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:44:53.858865 9149 catalog_manager.cc:1259] Loaded cluster ID: f15d3a63d6434470a3767cc9ec27fcca
I20260925 04:44:53.858929 9149 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:44:53.861258 9149 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:44:53.861943 9149 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b9ddd268e8b848aeaa70584a3e0391a4: Loaded TSK: 0
I20260925 04:44:53.863251 9149 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 04:44:53.942946 9139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:44:53.943212 9139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:44:53.943259 9139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:44:53.947414 9139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:44:53.947481 9139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:44:53.947572 9139 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260925 04:44:53.953498 9139 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:44821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/krb5kdc/oryx_127.0.98.65.keytab
--principal=oryx/127.0.98.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:34263
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=38575
--tserver_master_addrs=127.0.98.126:34911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9139
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:44:53.954759 9139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:44:53.955850 9139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:44:53.966020 9163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:53.966310 9139 server_base.cc:1034] running on GCE node
W20260925 04:44:53.966063 9164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:53.966571 9166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:53.966971 9139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:44:53.967700 9139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:44:53.968907 9139 hybrid_clock.cc:648] HybridClock initialized: now 1790311493968877 us; error 39 us; skew 500 ppm
Sep 25 04:44:53 dist-test-slave-kghz krb5kdc[8556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311493, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:44:53.972078 9139 init.cc:377] Logged in from keytab as oryx/127.0.98.65@KRBTEST.COM (short username oryx)
I20260925 04:44:53.973040 9139 webserver.cc:460] Webserver started at http://127.0.98.65:38575/ using document root <none> and password file <none>
I20260925 04:44:53.973587 9139 fs_manager.cc:359] Metadata directory not provided
I20260925 04:44:53.973661 9139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:44:53.977686 9139 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.002s
I20260925 04:44:53.980034 9173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:53.981365 9139 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 04:44:53.981506 9139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-0/wal
uuid: "74ff9ff0a144486c94e1554674c80577"
format_stamp: "Formatted at 2026-09-25 04:44:53 on dist-test-slave-kghz"
server_key: "ef382a7dcb7f4ac8cede597f294ceae8"
server_key_iv: "51e0d91831ee940f79b62e2235163d7e"
server_key_version: "encryptionkey@0"
I20260925 04:44:53.982103 9139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:44:54.007076 9139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:44:54.010545 9139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:44:54.010748 9139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:44:54.011372 9139 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:44:54.012253 9139 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:44:54.012321 9139 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:54.012426 9139 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:44:54.012465 9139 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:54.021826 9139 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:34263
I20260925 04:44:54.021844 9286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:34263 every 8 connection(s)
I20260925 04:44:54.022913 9139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
I20260925 04:44:54.028859 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 9139
I20260925 04:44:54.029320 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:35501
--local_ip_for_outbound_sockets=127.0.98.66
--tserver_master_addrs=127.0.98.126:34911
--webserver_port=40779
--webserver_interface=127.0.98.66
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:44821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.66
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/krb5kdc/oryx_127.0.98.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 04:44:54 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:44:54 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:44:54.036947 9290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:54.025025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34911 (local address 127.0.98.65:54831)
0925 04:44:54.025466 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:54.025529 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:54.026873 (+ 1344us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:54.027995 (+ 1122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:54.028002 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:54.028377 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:54.028868 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:54.028882 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:54.030301 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:54.030304 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:54.031198 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:54.031207 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:54.031975 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:54.031980 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:54.032172 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:54.032857 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:54.032884 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:54.035708 (+ 2824us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:34911: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":126,"threads_started":1}
W20260925 04:44:54.036984 9289 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:44:54.024949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:54831 (local address 127.0.98.126:34911)
0925 04:44:54.025414 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:54.025430 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:54.025644 (+ 214us) server_negotiation.cc:407] Connection header received
0925 04:44:54.027277 (+ 1633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:54.027289 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:54.027607 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:54.027893 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:54.029068 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:54.030136 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:54.031331 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:54.031810 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:54.031980 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:54.036034 (+ 4054us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.65:54831: BlockingRecv error: recv got EOF from 127.0.98.65:54831 (error 108)
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":113,"threads_started":1}
W20260925 04:44:54.037613 9287 heartbeater.cc:644] Failed to heartbeat to 127.0.98.126:34911 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.126:34911: Client connection negotiation failed: client connection to 127.0.98.126:34911: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Sep 25 04:44:54 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:44:54 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:44:54.044333 9290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:54.038120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34911 (local address 127.0.98.65:57007)
0925 04:44:54.038234 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:54.038286 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:54.038363 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:54.038720 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:54.038723 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:54.038881 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:54.039013 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:54.039019 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:54.040192 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:54.040196 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:54.040754 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:54.040763 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:54.041350 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:54.041353 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:54.041438 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:54.042067 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:54.042079 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:54.044187 (+ 2108us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:34911: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 04:44:54.044852 9289 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:44:54.038215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:57007 (local address 127.0.98.126:34911)
0925 04:44:54.038402 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:54.038409 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:54.038425 (+ 16us) server_negotiation.cc:407] Connection header received
0925 04:44:54.038487 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:54.038493 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:54.038555 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:54.038666 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:54.039139 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:54.040056 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:54.040865 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:54.041221 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:54.041324 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:54.044340 (+ 3016us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.65:57007: BlockingRecv error: recv got EOF from 127.0.98.65:57007 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
Sep 25 04:44:54 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:44:54 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:44:54.050164 9290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:54.044970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34911 (local address 127.0.98.65:47519)
0925 04:44:54.045297 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:54.045311 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:54.045416 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:54.045663 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:54.045665 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:54.045827 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:54.045898 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:54.045903 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:54.046672 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:54.046678 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:54.047206 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:54.047213 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:54.047645 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:54.047648 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:54.047729 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:54.048267 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:54.048280 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:54.050016 (+ 1736us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:34911: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":260}
W20260925 04:44:54.050298 9289 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:44:54.045047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:47519 (local address 127.0.98.126:34911)
0925 04:44:54.045175 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:54.045195 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:54.045374 (+ 179us) server_negotiation.cc:407] Connection header received
0925 04:44:54.045511 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:54.045517 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:54.045565 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:54.045649 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:54.046010 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:54.046568 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:54.047318 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:54.047540 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:54.047644 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:54.050144 (+ 2500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.65:47519: BlockingRecv error: recv got EOF from 127.0.98.65:47519 (error 108)
Metrics: {"server-negotiator.queue_time_us":54,"thread_start_us":118,"threads_started":1}
W20260925 04:44:54.050460 9287 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 04:44:54.145028 9291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:44:54.145313 9291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:44:54.145357 9291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:44:54.149500 9291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:44:54.149571 9291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:44:54.149673 9291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260925 04:44:54.155556 9291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:44821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/krb5kdc/oryx_127.0.98.66.keytab
--principal=oryx/127.0.98.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:35501
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=40779
--tserver_master_addrs=127.0.98.126:34911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9291
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:44:54.156663 9291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:44:54.157727 9291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:44:54.167667 9297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:54.167704 9291 server_base.cc:1034] running on GCE node
W20260925 04:44:54.167615 9300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:54.167615 9298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:54.168280 9291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:44:54.168917 9291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:44:54.170116 9291 hybrid_clock.cc:648] HybridClock initialized: now 1790311494170079 us; error 62 us; skew 500 ppm
Sep 25 04:44:54 dist-test-slave-kghz krb5kdc[8556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311494, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:44:54.172857 9291 init.cc:377] Logged in from keytab as oryx/127.0.98.66@KRBTEST.COM (short username oryx)
I20260925 04:44:54.173749 9291 webserver.cc:460] Webserver started at http://127.0.98.66:40779/ using document root <none> and password file <none>
I20260925 04:44:54.174448 9291 fs_manager.cc:359] Metadata directory not provided
I20260925 04:44:54.174549 9291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:44:54.178543 9291 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260925 04:44:54.180660 9307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:54.181921 9291 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 04:44:54.182098 9291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-1/wal
uuid: "bf746411a6fa48b7b94010f9d7783bbd"
format_stamp: "Formatted at 2026-09-25 04:44:53 on dist-test-slave-kghz"
server_key: "87bf5bdba37a7a4c21772523428f2bb2"
server_key_iv: "9347ef49c018ab28543bdc4170fa51cd"
server_key_version: "encryptionkey@0"
I20260925 04:44:54.182715 9291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:44:54.192046 9291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:44:54.195353 9291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:44:54.195554 9291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:44:54.196164 9291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:44:54.196992 9291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:44:54.197067 9291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:54.197151 9291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:44:54.197197 9291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:54.206445 9291 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:35501
I20260925 04:44:54.206516 9420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:35501 every 8 connection(s)
I20260925 04:44:54.207594 9291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
I20260925 04:44:54.216310 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 9291
I20260925 04:44:54.216846 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:44447
--local_ip_for_outbound_sockets=127.0.98.67
--tserver_master_addrs=127.0.98.126:34911
--webserver_port=43629
--webserver_interface=127.0.98.67
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:44821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.67
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/krb5kdc/oryx_127.0.98.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 04:44:54 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:44:54 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
W20260925 04:44:54.220340 9289 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:44:54.209723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:49177 (local address 127.0.98.126:34911)
0925 04:44:54.209902 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:54.209906 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:54.210892 (+ 986us) server_negotiation.cc:407] Connection header received
0925 04:44:54.212308 (+ 1416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:54.212311 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:54.212364 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:54.212462 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:54.213804 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:54.214478 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:54.215693 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:54.215923 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:54.216041 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:54.220160 (+ 4119us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.66:49177: BlockingRecv error: recv got EOF from 127.0.98.66:49177 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
I20260925 04:44:54.220870 9423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:54.210170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34911 (local address 127.0.98.66:49177)
0925 04:44:54.210715 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:54.210774 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:54.212064 (+ 1290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:54.212645 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:54.212653 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:54.213114 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:54.213633 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:54.213647 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:54.214615 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:54.214618 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:54.215554 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:54.215564 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:54.216123 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:54.216130 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:54.216360 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:54.217089 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:54.217115 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:54.219960 (+ 2845us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:34911: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":123,"threads_started":1}
W20260925 04:44:54.221575 9421 heartbeater.cc:644] Failed to heartbeat to 127.0.98.126:34911 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.126:34911: Client connection negotiation failed: client connection to 127.0.98.126:34911: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Sep 25 04:44:54 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:44:54 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:44:54.228088 9423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:54.222097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34911 (local address 127.0.98.66:44555)
0925 04:44:54.222220 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:54.222236 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:54.222351 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:54.222654 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:54.222656 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:54.222819 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:54.222916 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:54.222920 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:54.224054 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:54.224057 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:54.224605 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:54.224613 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:54.225246 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:54.225248 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:54.225319 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:54.225838 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:54.225849 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:54.227904 (+ 2055us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:34911: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":37}
W20260925 04:44:54.228125 9289 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:44:54.222211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:44555 (local address 127.0.98.126:34911)
0925 04:44:54.222389 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:54.222395 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:54.222414 (+ 19us) server_negotiation.cc:407] Connection header received
0925 04:44:54.222465 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:54.222470 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:54.222539 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:54.222642 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:54.223028 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:54.223929 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:54.224724 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:54.225130 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:54.225262 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:54.228032 (+ 2770us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.66:44555: BlockingRecv error: recv got EOF from 127.0.98.66:44555 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 25 04:44:54 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:44:54 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:44:54.234064 9423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:54.228710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34911 (local address 127.0.98.66:40445)
0925 04:44:54.228864 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:54.228877 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:54.228946 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:54.229168 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:54.229171 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:54.229332 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:54.229423 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:54.229429 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:54.230129 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:54.230132 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:54.231027 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:54.231036 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:54.231501 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:54.231504 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:54.231594 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:54.232065 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:54.232079 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:54.233934 (+ 1855us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:34911: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":91}
W20260925 04:44:54.234306 9421 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 04:44:54.234483 9289 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:44:54.228832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:40445 (local address 127.0.98.126:34911)
0925 04:44:54.228943 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:54.228946 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:54.228958 (+ 12us) server_negotiation.cc:407] Connection header received
0925 04:44:54.229029 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:54.229031 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:54.229067 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:54.229138 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:54.229527 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:54.230032 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:54.231172 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:54.231391 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:54.231483 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:54.234080 (+ 2597us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.66:40445: BlockingRecv error: recv got EOF from 127.0.98.66:40445 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260925 04:44:54.333438 9424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:44:54.333726 9424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:44:54.333809 9424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:44:54.337987 9424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:44:54.338073 9424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:44:54.338174 9424 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260925 04:44:54.343734 9424 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:44821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/krb5kdc/oryx_127.0.98.67.keytab
--principal=oryx/127.0.98.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:44447
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=43629
--tserver_master_addrs=127.0.98.126:34911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9424
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:44:54.344877 9424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:44:54.345939 9424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:44:54.355839 9430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:54.355830 9432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:44:54.355830 9429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:44:54.356192 9424 server_base.cc:1034] running on GCE node
I20260925 04:44:54.356594 9424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:44:54.357323 9424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:44:54.358474 9424 hybrid_clock.cc:648] HybridClock initialized: now 1790311494358440 us; error 52 us; skew 500 ppm
Sep 25 04:44:54 dist-test-slave-kghz krb5kdc[8556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311494, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:44:54.361182 9424 init.cc:377] Logged in from keytab as oryx/127.0.98.67@KRBTEST.COM (short username oryx)
I20260925 04:44:54.362130 9424 webserver.cc:460] Webserver started at http://127.0.98.67:43629/ using document root <none> and password file <none>
I20260925 04:44:54.362761 9424 fs_manager.cc:359] Metadata directory not provided
I20260925 04:44:54.362871 9424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:44:54.367055 9424 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 04:44:54.369208 9439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:54.370589 9424 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 04:44:54.370721 9424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-2/wal
uuid: "35dc94a4ad3541c99f3f738f0f0a5ef0"
format_stamp: "Formatted at 2026-09-25 04:44:53 on dist-test-slave-kghz"
server_key: "b7b783587a754635868474a5861e0e8c"
server_key_iv: "24a570e0d590d745215b630bc5f97ab6"
server_key_version: "encryptionkey@0"
I20260925 04:44:54.371277 9424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:44:54.387866 9424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:44:54.391177 9424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:44:54.391388 9424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:44:54.391990 9424 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:44:54.392858 9424 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:44:54.392930 9424 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:54.392989 9424 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:44:54.393018 9424 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:44:54.402447 9424 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:44447
I20260925 04:44:54.402510 9552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:44447 every 8 connection(s)
I20260925 04:44:54.403600 9424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
I20260925 04:44:54.404330 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 9424
Sep 25 04:44:54 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:44:54.415277 9289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:54.405198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44856 (local address 127.0.98.126:34911)
0925 04:44:54.405377 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:54.405381 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:54.405526 (+ 145us) server_negotiation.cc:407] Connection header received
0925 04:44:54.405730 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:54.405732 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:54.405776 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:54.405861 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:54.406810 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:54.407424 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:54.408684 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:54.408932 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:54.409007 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:54.410697 (+ 1690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:44:54.410703 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:44:54.410722 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:44:54.410753 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:44:54.412748 (+ 1995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:54.413318 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:54.413327 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:54.413336 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:54.413500 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:44:54.413814 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:44:54.413820 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:44:54.413825 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:44:54.414323 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:44:54.414543 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0925 04:44:54.414787 (+ 244us) server_negotiation.cc:299] Negotiation successful
0925 04:44:54.415036 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
Sep 25 04:44:54 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
W20260925 04:44:54.416884 9292 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:44:54.406027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:37585 (local address 127.0.98.126:34911)
0925 04:44:54.406158 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:54.406172 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:54.406910 (+ 738us) server_negotiation.cc:407] Connection header received
0925 04:44:54.408543 (+ 1633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:54.408546 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:54.408591 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:54.408692 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:54.410327 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:54.411013 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:54.412399 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:54.412767 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:54.412869 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:54.416769 (+ 3900us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.67:37585: BlockingRecv error: recv got EOF from 127.0.98.67:37585 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 04:44:54.417619 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:54.406167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34911 (local address 127.0.98.67:37585)
0925 04:44:54.406766 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:54.406827 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:54.408314 (+ 1487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:54.408863 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:54.408870 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:54.409403 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:54.410092 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:54.410117 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:54.411151 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:54.411156 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:54.412279 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:54.412288 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:54.412921 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:54.412928 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:54.413132 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:54.413736 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:54.413752 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:54.416623 (+ 2871us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:34911: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":126,"threads_started":1}
W20260925 04:44:54.418165 9553 heartbeater.cc:644] Failed to heartbeat to 127.0.98.126:34911 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.126:34911: Client connection negotiation failed: client connection to 127.0.98.126:34911: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Sep 25 04:44:54 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:44:54 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:44:54.423600 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:54.418664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34911 (local address 127.0.98.67:59753)
0925 04:44:54.418846 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:54.418867 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:54.418965 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:54.419170 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:54.419172 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:54.419308 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:54.419401 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:54.419405 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:54.420068 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:54.420070 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:54.420636 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:54.420643 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:54.421066 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:54.421070 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:54.421148 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:54.421587 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:54.421598 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:54.423493 (+ 1895us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:34911: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":39}
W20260925 04:44:54.423744 9292 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:44:54.418743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:59753 (local address 127.0.98.126:34911)
0925 04:44:54.418850 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:54.418854 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:54.418925 (+ 71us) server_negotiation.cc:407] Connection header received
0925 04:44:54.419040 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:54.419042 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:54.419081 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:54.419145 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:54.419489 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:54.419983 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:54.420728 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:54.420943 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:54.421020 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:54.423628 (+ 2608us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.67:59753: BlockingRecv error: recv got EOF from 127.0.98.67:59753 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 25 04:44:54 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:44:54 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:44:54.429365 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:54.424219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34911 (local address 127.0.98.67:51645)
0925 04:44:54.424336 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:54.424352 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:54.424438 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:54.424699 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:54.424701 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:54.424854 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:54.424925 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:54.424930 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:54.425705 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:54.425711 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:54.426241 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:54.426304 (+ 63us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:54.426729 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:54.426733 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:54.426813 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:54.427330 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:54.427342 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:54.429221 (+ 1879us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:34911: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 04:44:54.429610 9553 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 04:44:54.429644 9292 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:44:54.424308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:51645 (local address 127.0.98.126:34911)
0925 04:44:54.424452 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:54.424458 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:54.424476 (+ 18us) server_negotiation.cc:407] Connection header received
0925 04:44:54.424553 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:54.424556 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:54.424602 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:54.424684 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:54.425089 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:54.425600 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:54.426406 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:54.426625 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:54.426728 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:54.429377 (+ 2649us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.67:51645: BlockingRecv error: recv got EOF from 127.0.98.67:51645 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 25 04:44:55 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:44:55 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:44:55.057255 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:55.051182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34911 (local address 127.0.98.65:48679)
0925 04:44:55.051477 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:55.051500 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:55.051643 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:55.052001 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:55.052004 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:55.052244 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:55.052378 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:55.052386 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:55.053211 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:55.053214 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:55.053940 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:55.053951 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:55.054546 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:55.054552 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:55.054633 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:55.055226 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:55.055243 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:55.057118 (+ 1875us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:34911: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":87,"threads_started":1}
W20260925 04:44:55.057502 9558 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:44:55.051334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:48679 (local address 127.0.98.126:34911)
0925 04:44:55.051585 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:55.051590 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:55.051609 (+ 19us) server_negotiation.cc:407] Connection header received
0925 04:44:55.051802 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:55.051805 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:55.051865 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:55.051967 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:55.052516 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:55.053078 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:55.054061 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:55.054336 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:55.054518 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:55.057319 (+ 2801us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.65:48679: BlockingRecv error: recv got EOF from 127.0.98.65:48679 (error 108)
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":71,"threads_started":1}
Sep 25 04:44:55 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:44:55 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:44:55.240433 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:55.234953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34911 (local address 127.0.98.66:49327)
0925 04:44:55.235261 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:55.235275 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:55.235368 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:55.235621 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:55.235623 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:55.235793 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:55.235884 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:55.235889 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:55.236724 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:55.236726 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:55.237318 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:55.237326 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:55.237739 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:55.237742 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:55.237801 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:55.238466 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:55.238478 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:55.240303 (+ 1825us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:34911: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":74,"threads_started":1}
W20260925 04:44:55.240625 9558 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:44:55.235137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:49327 (local address 127.0.98.126:34911)
0925 04:44:55.235307 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:55.235311 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:55.235325 (+ 14us) server_negotiation.cc:407] Connection header received
0925 04:44:55.235454 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:55.235462 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:55.235504 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:55.235574 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:55.236063 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:55.236619 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:55.237422 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:55.237643 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:55.237724 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:55.240486 (+ 2762us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.66:49327: BlockingRecv error: recv got EOF from 127.0.98.66:49327 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 25 04:44:55 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:44:55 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:44:55.436218 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:55.430397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34911 (local address 127.0.98.67:41691)
0925 04:44:55.430654 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:55.430668 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:55.430770 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:55.431063 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:55.431065 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:55.431243 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:55.431361 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:55.431366 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:55.432114 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:55.432117 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:55.432719 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:55.432727 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:55.433212 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:55.433215 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:55.433287 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:55.433769 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:55.433784 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:55.435948 (+ 2164us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:34911: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":90,"threads_started":1}
W20260925 04:44:55.436225 9558 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:44:55.430452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:41691 (local address 127.0.98.126:34911)
0925 04:44:55.430588 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:55.430592 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:55.430758 (+ 166us) server_negotiation.cc:407] Connection header received
0925 04:44:55.430890 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:55.430893 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:55.430943 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:55.431032 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:55.431457 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:55.432017 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:55.432826 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:55.433089 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:55.433173 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:55.436092 (+ 2919us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.67:41691: BlockingRecv error: recv got EOF from 127.0.98.67:41691 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 25 04:44:56 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:44:56 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:44:56.064589 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:56.058173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34911 (local address 127.0.98.65:50363)
0925 04:44:56.058486 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:56.058499 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:56.058625 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:56.058893 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:56.058895 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:56.059083 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:56.059177 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:56.059182 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:56.059951 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:56.059953 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:56.060498 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:56.060505 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:56.060983 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:56.060987 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:56.061078 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:56.061738 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:56.061756 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:56.064361 (+ 2605us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:34911: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":142,"threads_started":1}
W20260925 04:44:56.064755 9562 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:44:56.058224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:50363 (local address 127.0.98.126:34911)
0925 04:44:56.058497 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:56.058501 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:56.058590 (+ 89us) server_negotiation.cc:407] Connection header received
0925 04:44:56.058711 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:56.058713 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:56.058759 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:56.058863 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:56.059304 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:56.059855 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:56.060620 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:56.060862 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:56.060953 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:56.064575 (+ 3622us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.65:50363: BlockingRecv error: recv got EOF from 127.0.98.65:50363 (error 108)
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":105,"threads_started":1}
Sep 25 04:44:56 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:44:56 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:44:56.246691 9563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:56.241294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34911 (local address 127.0.98.66:44581)
0925 04:44:56.241560 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:56.241593 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:56.241692 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:56.241956 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:56.241959 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:56.242155 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:56.242288 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:56.242295 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:56.243083 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:56.243086 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:56.243680 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:56.243689 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:56.244113 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:56.244116 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:56.244175 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:56.244702 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:56.244714 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:56.246557 (+ 1843us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:34911: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":90,"threads_started":1}
W20260925 04:44:56.246946 9562 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:44:56.241299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:44581 (local address 127.0.98.126:34911)
0925 04:44:56.241453 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:56.241458 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:56.241668 (+ 210us) server_negotiation.cc:407] Connection header received
0925 04:44:56.241764 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:56.241766 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:56.241812 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:56.241893 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:56.242411 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:56.242967 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:56.243802 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:56.244017 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:56.244109 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:56.246694 (+ 2585us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.66:44581: BlockingRecv error: recv got EOF from 127.0.98.66:44581 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 25 04:44:56 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:44:56 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:44:56.447348 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:56.437110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34911 (local address 127.0.98.67:51777)
0925 04:44:56.437436 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:56.437454 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:56.437600 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:56.437869 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:56.437872 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:56.438085 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:56.438177 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:56.438182 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:56.438998 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:56.439005 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:56.439583 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:56.439590 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:56.440036 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:56.440048 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:56.440134 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:56.444945 (+ 4811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:56.444959 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:56.447142 (+ 2183us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:34911: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":105,"threads_started":1}
W20260925 04:44:56.447403 9562 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:44:56.437296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:51777 (local address 127.0.98.126:34911)
0925 04:44:56.437459 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:56.437464 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:56.437535 (+ 71us) server_negotiation.cc:407] Connection header received
0925 04:44:56.437704 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:56.437708 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:56.437762 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:56.437847 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:56.438333 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:56.438867 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:56.439711 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:56.439929 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:56.440033 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:56.447291 (+ 7258us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.67:51777: BlockingRecv error: recv got EOF from 127.0.98.67:51777 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 25 04:44:57 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:44:57 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:44:57.072731 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:57.066646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34911 (local address 127.0.98.65:42299)
0925 04:44:57.066940 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:57.066954 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:57.067087 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:57.067404 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:57.067407 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:57.067673 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:57.067811 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:57.067821 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:57.068723 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:57.068730 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:57.069327 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:57.069336 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:57.069827 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:57.069830 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:57.069902 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:57.070507 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:57.070523 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:57.072590 (+ 2067us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:34911: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":107,"threads_started":1}
W20260925 04:44:57.072914 9566 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:44:57.066735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:42299 (local address 127.0.98.126:34911)
0925 04:44:57.067045 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:57.067049 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:57.067065 (+ 16us) server_negotiation.cc:407] Connection header received
0925 04:44:57.067196 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:57.067198 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:57.067243 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:57.067368 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:57.067950 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:57.068597 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:57.069441 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:57.069676 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:57.069757 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:57.072766 (+ 3009us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.65:42299: BlockingRecv error: recv got EOF from 127.0.98.65:42299 (error 108)
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":147,"threads_started":1}
Sep 25 04:44:57 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:44:57 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:44:57.253438 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:57.247523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34911 (local address 127.0.98.66:53051)
0925 04:44:57.247830 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:57.247844 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:57.247997 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:57.248282 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:57.248285 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:57.248557 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:57.248684 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:57.248690 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:57.249444 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:57.249450 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:57.250030 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:57.250038 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:57.250566 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:57.250570 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:57.250651 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:57.251272 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:57.251285 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:57.253220 (+ 1935us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:34911: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":67,"threads_started":1}
W20260925 04:44:57.253458 9566 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:44:57.247621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:53051 (local address 127.0.98.126:34911)
0925 04:44:57.247789 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:57.247794 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:57.247945 (+ 151us) server_negotiation.cc:407] Connection header received
0925 04:44:57.248109 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:57.248115 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:57.248166 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:57.248258 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:57.248801 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:57.249330 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:57.250154 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:57.250449 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:57.250566 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:57.253356 (+ 2790us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.66:53051: BlockingRecv error: recv got EOF from 127.0.98.66:53051 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 25 04:44:57 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:44:57 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:44:57.454867 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:57.448439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34911 (local address 127.0.98.67:44679)
0925 04:44:57.448871 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:57.448885 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:57.449017 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:57.449363 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:57.449366 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:57.449557 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:57.449656 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:57.449662 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:57.450441 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:57.450444 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:57.451023 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:57.451031 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:57.451533 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:57.451537 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:57.451638 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:57.452316 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:57.452328 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:57.454680 (+ 2352us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:34911: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":97,"threads_started":1}
W20260925 04:44:57.454962 9566 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:44:57.448532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:44679 (local address 127.0.98.126:34911)
0925 04:44:57.448728 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:57.448733 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:57.449014 (+ 281us) server_negotiation.cc:407] Connection header received
0925 04:44:57.449124 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:57.449127 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:57.449180 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:57.449268 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:57.449778 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:57.450334 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:57.451165 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:57.451393 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:57.451476 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:57.454842 (+ 3366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.67:44679: BlockingRecv error: recv got EOF from 127.0.98.67:44679 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 25 04:44:58 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:44:58 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:44:58.079726 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:58.073649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34911 (local address 127.0.98.65:51743)
0925 04:44:58.073973 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:58.073988 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:58.074149 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:58.074492 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:58.074494 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:58.074694 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:58.074790 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:58.074795 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:58.075600 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:58.075605 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:58.076442 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:58.076460 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:58.077095 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:58.077098 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:58.077165 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:58.077708 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:58.077720 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:58.079612 (+ 1892us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:34911: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":87,"threads_started":1}
W20260925 04:44:58.079913 9570 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:44:58.073797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:51743 (local address 127.0.98.126:34911)
0925 04:44:58.074052 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:58.074055 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:58.074071 (+ 16us) server_negotiation.cc:407] Connection header received
0925 04:44:58.074280 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:58.074283 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:58.074357 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:58.074458 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:58.074905 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:58.075483 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:58.076636 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:58.076973 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:58.077056 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:58.079768 (+ 2712us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.65:51743: BlockingRecv error: recv got EOF from 127.0.98.65:51743 (error 108)
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":57,"threads_started":1}
Sep 25 04:44:58 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:44:58 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:44:58.260197 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:58.254570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34911 (local address 127.0.98.66:60571)
0925 04:44:58.254823 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:58.254844 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:58.254947 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:58.255192 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:58.255194 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:58.255369 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:58.255483 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:58.255489 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:58.256513 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:58.256517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:58.257091 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:58.257099 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:58.257553 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:58.257557 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:58.257641 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:58.258136 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:58.258147 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:58.260042 (+ 1895us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:34911: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":96,"threads_started":1}
W20260925 04:44:58.260319 9570 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:44:58.254649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:60571 (local address 127.0.98.126:34911)
0925 04:44:58.254788 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:58.254793 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:58.254913 (+ 120us) server_negotiation.cc:407] Connection header received
0925 04:44:58.255029 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:58.255032 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:58.255079 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:58.255163 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:58.255683 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:58.256404 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:58.257211 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:58.257438 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:58.257542 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:58.260212 (+ 2670us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.66:60571: BlockingRecv error: recv got EOF from 127.0.98.66:60571 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 25 04:44:58 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:44:58 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:44:58.461810 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:58.455719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34911 (local address 127.0.98.67:50445)
0925 04:44:58.456162 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:58.456175 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:58.456319 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:58.456628 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:58.456632 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:58.456852 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:58.456962 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:58.456968 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:58.457740 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:58.457746 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:58.458354 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:58.458362 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:58.458814 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:58.458818 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:58.458906 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:58.459518 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:58.459533 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:58.461679 (+ 2146us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:34911: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":109,"threads_started":1}
W20260925 04:44:58.461946 9570 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:44:58.455862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:50445 (local address 127.0.98.126:34911)
0925 04:44:58.456071 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:58.456076 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:58.456264 (+ 188us) server_negotiation.cc:407] Connection header received
0925 04:44:58.456454 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:58.456457 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:58.456504 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:58.456609 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:58.457094 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:58.457627 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:58.458475 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:58.458700 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:58.458813 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:58.461806 (+ 2993us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.67:50445: BlockingRecv error: recv got EOF from 127.0.98.67:50445 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
Sep 25 04:44:59 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:44:59 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:44:59.087407 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:59.080673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34911 (local address 127.0.98.65:53705)
0925 04:44:59.081023 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:59.081038 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:59.081170 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:59.081448 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:59.081450 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:59.081640 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:59.081734 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:59.081739 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:59.082617 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:59.082620 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:59.083218 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:59.083228 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:59.083741 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:59.083743 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:59.083811 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:59.084581 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:59.084593 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:59.087217 (+ 2624us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:34911: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":119,"threads_started":1}
W20260925 04:44:59.087491 9574 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:44:59.080851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:53705 (local address 127.0.98.126:34911)
0925 04:44:59.081138 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:59.081142 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:59.081158 (+ 16us) server_negotiation.cc:407] Connection header received
0925 04:44:59.081271 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:59.081274 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:59.081321 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:59.081410 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:59.081877 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:59.082494 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:59.083354 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:59.083614 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:59.083716 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:59.087355 (+ 3639us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.65:53705: BlockingRecv error: recv got EOF from 127.0.98.65:53705 (error 108)
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":91,"threads_started":1}
Sep 25 04:44:59 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:44:59 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:44:59.267226 9575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:59.261288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34911 (local address 127.0.98.66:59681)
0925 04:44:59.261655 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:59.261673 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:59.261825 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:59.262158 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:59.262161 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:59.262433 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:59.262532 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:59.262537 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:59.263311 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:59.263318 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:59.263903 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:59.263911 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:59.264375 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:59.264378 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:59.264472 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:59.265045 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:59.265057 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:59.267034 (+ 1977us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:34911: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":121,"threads_started":1}
W20260925 04:44:59.267282 9574 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:44:59.261493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:59681 (local address 127.0.98.126:34911)
0925 04:44:59.261670 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:59.261675 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:59.261761 (+ 86us) server_negotiation.cc:407] Connection header received
0925 04:44:59.261939 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:59.261943 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:59.262036 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:59.262137 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:59.262655 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:59.263199 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:59.264031 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:59.264260 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:59.264370 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:59.267174 (+ 2804us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.66:59681: BlockingRecv error: recv got EOF from 127.0.98.66:59681 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 25 04:44:59 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:44:59 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:44:59.468990 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:44:59.462731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34911 (local address 127.0.98.67:50521)
0925 04:44:59.463152 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0925 04:44:59.463167 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:44:59.463306 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:44:59.463633 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:44:59.463636 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:44:59.463875 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:44:59.463977 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:59.463983 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:59.464830 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:59.464834 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:59.465492 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:44:59.465501 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:44:59.466011 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:44:59.466015 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:44:59.466108 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:59.466758 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:44:59.466771 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:44:59.468796 (+ 2025us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:34911: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":92,"threads_started":1}
W20260925 04:44:59.469043 9574 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:44:59.462867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:50521 (local address 127.0.98.126:34911)
0925 04:44:59.463057 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 04:44:59.463062 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:44:59.463256 (+ 194us) server_negotiation.cc:407] Connection header received
0925 04:44:59.463421 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:44:59.463425 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:44:59.463509 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:44:59.463609 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:44:59.464102 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:59.464699 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:59.465620 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:44:59.465880 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:44:59.466002 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:44:59.468936 (+ 2934us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.67:50521: BlockingRecv error: recv got EOF from 127.0.98.67:50521 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 25 04:45:00 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:45:00 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:45:00.094805 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:45:00.088373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34911 (local address 127.0.98.65:55809)
0925 04:45:00.088689 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0925 04:45:00.088707 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 04:45:00.088823 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:45:00.089106 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:45:00.089109 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:45:00.089307 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:45:00.089407 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:45:00.089412 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:45:00.090364 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:45:00.090368 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:45:00.091090 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:45:00.091104 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:45:00.091725 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:45:00.091729 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:45:00.091818 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:00.092380 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:45:00.092393 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:45:00.094679 (+ 2286us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:34911: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":74,"threads_started":1}
W20260925 04:45:00.095039 9578 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:45:00.088457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:55809 (local address 127.0.98.126:34911)
0925 04:45:00.088757 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0925 04:45:00.088761 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:45:00.088777 (+ 16us) server_negotiation.cc:407] Connection header received
0925 04:45:00.088918 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:45:00.088921 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:45:00.088966 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:45:00.089049 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:45:00.089509 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:00.090175 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:00.091251 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:00.091563 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:00.091673 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:00.094872 (+ 3199us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.65:55809: BlockingRecv error: recv got EOF from 127.0.98.65:55809 (error 108)
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":109,"threads_started":1}
Sep 25 04:45:00 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:45:00 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:45:00.274642 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:45:00.268260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34911 (local address 127.0.98.66:38877)
0925 04:45:00.268643 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0925 04:45:00.268660 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 04:45:00.268914 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:45:00.269239 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:45:00.269242 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:45:00.269485 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:45:00.269577 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:45:00.269582 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:45:00.270505 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:45:00.270512 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:45:00.271355 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:45:00.271367 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:45:00.271960 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:45:00.271963 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:45:00.272028 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:00.272568 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:45:00.272580 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:45:00.274526 (+ 1946us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:34911: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":100,"threads_started":1}
W20260925 04:45:00.274848 9578 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:45:00.268410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:38877 (local address 127.0.98.126:34911)
0925 04:45:00.268621 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0925 04:45:00.268626 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:45:00.268842 (+ 216us) server_negotiation.cc:407] Connection header received
0925 04:45:00.269065 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:45:00.269072 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:45:00.269119 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:45:00.269216 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:45:00.269700 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:00.270378 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:00.271515 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:00.271848 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:00.271935 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:00.274673 (+ 2738us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.66:38877: BlockingRecv error: recv got EOF from 127.0.98.66:38877 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 25 04:45:00 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:45:00 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:45:00.475766 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:45:00.469896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34911 (local address 127.0.98.67:47125)
0925 04:45:00.470206 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0925 04:45:00.470220 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:45:00.470350 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:45:00.470603 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:45:00.470606 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:45:00.470788 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:45:00.470889 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:45:00.470894 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:45:00.471676 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:45:00.471679 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:45:00.472254 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:45:00.472262 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:45:00.472750 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:45:00.472753 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:45:00.472817 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:00.473309 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:45:00.473321 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:45:00.475586 (+ 2265us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:34911: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":136,"threads_started":1}
W20260925 04:45:00.475826 9578 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:45:00.470045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:47125 (local address 127.0.98.126:34911)
0925 04:45:00.470184 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 04:45:00.470187 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:45:00.470345 (+ 158us) server_negotiation.cc:407] Connection header received
0925 04:45:00.470430 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:45:00.470432 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:45:00.470490 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:45:00.470569 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:45:00.471019 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:00.471580 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:00.472399 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:00.472641 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:00.472731 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:00.475719 (+ 2988us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.67:47125: BlockingRecv error: recv got EOF from 127.0.98.67:47125 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 25 04:45:01 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:45:01 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:45:01.101727 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:45:01.095699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34911 (local address 127.0.98.65:52931)
0925 04:45:01.096170 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0925 04:45:01.096184 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:45:01.096316 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:45:01.096645 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:45:01.096648 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:45:01.096856 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:45:01.096960 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:45:01.096966 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:45:01.097786 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:45:01.097791 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:45:01.098454 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:45:01.098463 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:45:01.098930 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:45:01.098934 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:45:01.099021 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:01.099545 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:45:01.099558 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:45:01.101603 (+ 2045us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:34911: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":81,"threads_started":1}
W20260925 04:45:01.101943 9582 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:45:01.095755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:52931 (local address 127.0.98.126:34911)
0925 04:45:01.096023 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0925 04:45:01.096027 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:45:01.096276 (+ 249us) server_negotiation.cc:407] Connection header received
0925 04:45:01.096418 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:45:01.096422 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:45:01.096484 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:45:01.096597 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:45:01.097077 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:01.097671 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:01.098574 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:01.098816 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:01.098926 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:01.101760 (+ 2834us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.65:52931: BlockingRecv error: recv got EOF from 127.0.98.65:52931 (error 108)
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":80,"threads_started":1}
Sep 25 04:45:01 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:45:01 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:45:01.282191 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:45:01.275595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34911 (local address 127.0.98.66:46521)
0925 04:45:01.275916 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0925 04:45:01.275931 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:45:01.276080 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:45:01.276449 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:45:01.276452 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:45:01.276734 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:45:01.276864 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:45:01.276871 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:45:01.277865 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:45:01.277869 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:45:01.278529 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:45:01.278538 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:45:01.279124 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:45:01.279128 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:45:01.279218 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:01.279777 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:45:01.279789 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:45:01.282003 (+ 2214us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:34911: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":94,"threads_started":1}
W20260925 04:45:01.282300 9582 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:45:01.275719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:46521 (local address 127.0.98.126:34911)
0925 04:45:01.275885 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 04:45:01.275890 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:45:01.276034 (+ 144us) server_negotiation.cc:407] Connection header received
0925 04:45:01.276240 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:45:01.276244 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:45:01.276301 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:45:01.276403 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:45:01.277000 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:01.277748 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:01.278644 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:01.278894 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:01.279079 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:01.282155 (+ 3076us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.66:46521: BlockingRecv error: recv got EOF from 127.0.98.66:46521 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 25 04:45:01 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:45:01 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:45:01.487594 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:45:01.476523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34911 (local address 127.0.98.67:59933)
0925 04:45:01.476906 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0925 04:45:01.476920 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:45:01.477088 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:45:01.477478 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:45:01.477481 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:45:01.477723 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:45:01.477834 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:45:01.477840 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:45:01.478720 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:45:01.478725 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:45:01.479292 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:45:01.479299 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:45:01.479819 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:45:01.479823 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:45:01.479911 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:01.484783 (+ 4872us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:45:01.484797 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:45:01.487393 (+ 2596us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:34911: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":88,"threads_started":1}
W20260925 04:45:01.487668 9582 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:45:01.476679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:59933 (local address 127.0.98.126:34911)
0925 04:45:01.476847 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 04:45:01.476852 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:45:01.477017 (+ 165us) server_negotiation.cc:407] Connection header received
0925 04:45:01.477228 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:45:01.477232 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:45:01.477307 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:45:01.477429 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:45:01.477963 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:01.478610 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:01.479414 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:01.479704 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:01.479815 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:01.487553 (+ 7738us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.67:59933: BlockingRecv error: recv got EOF from 127.0.98.67:59933 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 25 04:45:02 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:45:02 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:45:02.108278 9585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:45:02.102500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34911 (local address 127.0.98.65:42071)
0925 04:45:02.102777 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0925 04:45:02.102791 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:45:02.102879 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:45:02.103229 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:45:02.103231 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:45:02.103426 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:45:02.103557 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:45:02.103564 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:45:02.104466 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:45:02.104469 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:45:02.105060 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:45:02.105068 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:45:02.105516 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:45:02.105519 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:45:02.105584 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:02.106218 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:45:02.106230 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:45:02.108159 (+ 1929us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:34911: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":106,"threads_started":1}
W20260925 04:45:02.108464 9586 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:45:02.102739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:42071 (local address 127.0.98.126:34911)
0925 04:45:02.102969 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0925 04:45:02.102973 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:45:02.102984 (+ 11us) server_negotiation.cc:407] Connection header received
0925 04:45:02.103018 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:45:02.103021 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:45:02.103062 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:45:02.103155 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:45:02.103718 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:02.104345 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:02.105163 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:02.105399 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:02.105477 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:02.108307 (+ 2830us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.65:42071: BlockingRecv error: recv got EOF from 127.0.98.65:42071 (error 108)
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":84,"threads_started":1}
Sep 25 04:45:02 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:45:02 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:45:02.288842 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:45:02.283034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34911 (local address 127.0.98.66:55331)
0925 04:45:02.283289 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0925 04:45:02.283302 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:45:02.283401 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:45:02.283824 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:45:02.283826 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:45:02.284018 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:45:02.284117 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:45:02.284122 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:45:02.285029 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:45:02.285031 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:45:02.285606 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:45:02.285615 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:45:02.286152 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:45:02.286157 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:45:02.286230 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:02.286806 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:45:02.286818 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:45:02.288722 (+ 1904us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:34911: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":92,"threads_started":1}
W20260925 04:45:02.289052 9586 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:45:02.283145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:55331 (local address 127.0.98.126:34911)
0925 04:45:02.283281 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 04:45:02.283285 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:45:02.283370 (+ 85us) server_negotiation.cc:407] Connection header received
0925 04:45:02.283478 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:45:02.283480 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:45:02.283524 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:45:02.283616 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:45:02.284243 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:02.284929 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:02.285724 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:02.285943 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:02.286089 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:02.288889 (+ 2800us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.66:55331: BlockingRecv error: recv got EOF from 127.0.98.66:55331 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 25 04:45:02 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:45:02 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:45:02.494032 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:45:02.488441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34911 (local address 127.0.98.67:58845)
0925 04:45:02.488795 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0925 04:45:02.488808 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:45:02.488942 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:45:02.489220 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:45:02.489223 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:45:02.489439 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:45:02.489529 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:45:02.489534 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:45:02.490322 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:45:02.490325 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:45:02.490894 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:45:02.490902 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:45:02.491352 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:45:02.491356 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:45:02.491442 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:02.492008 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:45:02.492020 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:45:02.493903 (+ 1883us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:34911: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":102,"threads_started":1}
W20260925 04:45:02.494159 9586 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:45:02.488557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:58845 (local address 127.0.98.126:34911)
0925 04:45:02.488727 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 04:45:02.488732 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:45:02.488883 (+ 151us) server_negotiation.cc:407] Connection header received
0925 04:45:02.489042 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:45:02.489046 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:45:02.489103 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:45:02.489197 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:45:02.489655 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:02.490180 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:02.491013 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:02.491243 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:02.491349 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:02.494037 (+ 2688us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.67:58845: BlockingRecv error: recv got EOF from 127.0.98.67:58845 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 25 04:45:03 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:45:03 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:45:03.115115 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:45:03.109148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34911 (local address 127.0.98.65:33253)
0925 04:45:03.109435 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0925 04:45:03.109457 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 04:45:03.109554 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:45:03.109855 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:45:03.109857 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:45:03.110052 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:45:03.110176 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:45:03.110182 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:45:03.110981 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:45:03.110984 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:45:03.111567 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:45:03.111575 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:45:03.112084 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:45:03.112087 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:45:03.112186 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:03.112775 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:45:03.112787 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:45:03.114936 (+ 2149us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:34911: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":108,"threads_started":1}
W20260925 04:45:03.115253 9590 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:45:03.109281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:33253 (local address 127.0.98.126:34911)
0925 04:45:03.109516 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0925 04:45:03.109519 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:45:03.109533 (+ 14us) server_negotiation.cc:407] Connection header received
0925 04:45:03.109651 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:45:03.109653 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:45:03.109703 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:45:03.109805 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:45:03.110326 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:03.110877 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:03.111729 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:03.111963 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:03.112053 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:03.115076 (+ 3023us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.65:33253: BlockingRecv error: recv got EOF from 127.0.98.65:33253 (error 108)
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":92,"threads_started":1}
Sep 25 04:45:03 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:45:03 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:45:03.295590 9591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:45:03.289717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34911 (local address 127.0.98.66:40035)
0925 04:45:03.290081 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0925 04:45:03.290095 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:45:03.290239 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:45:03.290534 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:45:03.290537 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:45:03.290758 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:45:03.290848 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:45:03.290854 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:45:03.291628 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:45:03.291635 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:45:03.292194 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:45:03.292201 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:45:03.292672 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:45:03.292675 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:45:03.292756 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:03.293326 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:45:03.293338 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:45:03.295470 (+ 2132us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:34911: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":86,"threads_started":1}
W20260925 04:45:03.295796 9590 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:45:03.289846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:40035 (local address 127.0.98.126:34911)
0925 04:45:03.290029 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 04:45:03.290034 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:45:03.290175 (+ 141us) server_negotiation.cc:407] Connection header received
0925 04:45:03.290374 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:45:03.290377 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:45:03.290430 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:45:03.290514 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:45:03.290963 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:03.291509 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:03.292313 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:03.292553 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:03.292668 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:03.295639 (+ 2971us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.66:40035: BlockingRecv error: recv got EOF from 127.0.98.66:40035 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 25 04:45:03 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:45:03 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:45:03.500327 9592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:45:03.495009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34911 (local address 127.0.98.67:48393)
0925 04:45:03.495283 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0925 04:45:03.495296 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:45:03.495404 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:45:03.495643 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:45:03.495646 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:45:03.495838 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:45:03.495931 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:45:03.495936 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:45:03.496645 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:45:03.496647 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:45:03.497262 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:45:03.497270 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:45:03.497691 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:45:03.497694 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:45:03.497758 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:03.498310 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:45:03.498322 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:45:03.500200 (+ 1878us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:34911: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":108,"threads_started":1}
W20260925 04:45:03.500586 9590 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:45:03.495134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:48393 (local address 127.0.98.126:34911)
0925 04:45:03.495265 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 04:45:03.495269 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:45:03.495393 (+ 124us) server_negotiation.cc:407] Connection header received
0925 04:45:03.495476 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:45:03.495479 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:45:03.495529 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:45:03.495602 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:45:03.496033 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:03.496550 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:03.497366 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:03.497587 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:03.497665 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:03.500464 (+ 2799us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.67:48393: BlockingRecv error: recv got EOF from 127.0.98.67:48393 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 25 04:45:04 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:45:04 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:45:04.121776 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:45:04.116205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34911 (local address 127.0.98.65:42775)
0925 04:45:04.116515 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0925 04:45:04.116533 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 04:45:04.116636 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:45:04.116929 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:45:04.116932 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:45:04.117175 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:45:04.117271 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:45:04.117277 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:45:04.118123 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:45:04.118133 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:45:04.118720 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:45:04.118728 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:45:04.119157 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:45:04.119160 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:45:04.119225 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:04.119715 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:45:04.119727 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:45:04.121637 (+ 1910us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:34911: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":93,"threads_started":1}
W20260925 04:45:04.121960 9594 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:45:04.116338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:42775 (local address 127.0.98.126:34911)
0925 04:45:04.116625 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0925 04:45:04.116632 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:45:04.116670 (+ 38us) server_negotiation.cc:407] Connection header received
0925 04:45:04.116720 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:45:04.116723 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:45:04.116782 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:45:04.116873 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:45:04.117381 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:04.118018 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:04.118817 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:04.119035 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:04.119113 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:04.121787 (+ 2674us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.65:42775: BlockingRecv error: recv got EOF from 127.0.98.65:42775 (error 108)
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":98,"threads_started":1}
Sep 25 04:45:04 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:45:04 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:45:04.302220 9595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:45:04.296531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34911 (local address 127.0.98.66:36801)
0925 04:45:04.296805 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0925 04:45:04.296819 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:45:04.296948 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:45:04.297261 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:45:04.297264 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:45:04.297473 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:45:04.297566 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:45:04.297571 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:45:04.298375 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:45:04.298380 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:45:04.298950 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:45:04.298959 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:45:04.299424 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:45:04.299428 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:45:04.299508 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:04.300082 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:45:04.300094 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:45:04.302045 (+ 1951us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:34911: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":79,"threads_started":1}
W20260925 04:45:04.302371 9594 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:45:04.296595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:36801 (local address 127.0.98.126:34911)
0925 04:45:04.296748 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 04:45:04.296751 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:45:04.296904 (+ 153us) server_negotiation.cc:407] Connection header received
0925 04:45:04.297069 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:45:04.297073 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:45:04.297122 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:45:04.297237 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:45:04.297690 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:04.298225 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:04.299076 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:04.299312 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:04.299417 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:04.302224 (+ 2807us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.66:36801: BlockingRecv error: recv got EOF from 127.0.98.66:36801 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 25 04:45:04 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:45:04 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:45:04.507332 9596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:45:04.501134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34911 (local address 127.0.98.67:56687)
0925 04:45:04.501490 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0925 04:45:04.501507 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 04:45:04.501636 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:45:04.501915 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:45:04.501918 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:45:04.502130 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:45:04.502226 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:45:04.502231 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:45:04.503110 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:45:04.503113 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:45:04.504084 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:45:04.504096 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:45:04.504658 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:45:04.504661 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:45:04.504749 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:04.505253 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:45:04.505265 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:45:04.507209 (+ 1944us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:34911: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":101,"threads_started":1}
W20260925 04:45:04.507547 9594 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:45:04.501346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:56687 (local address 127.0.98.126:34911)
0925 04:45:04.501500 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 04:45:04.501507 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:45:04.501572 (+ 65us) server_negotiation.cc:407] Connection header received
0925 04:45:04.501744 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:45:04.501748 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:45:04.501805 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:45:04.501894 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:45:04.502406 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:04.502946 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:04.504225 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:04.504542 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:04.504654 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:04.507353 (+ 2699us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.67:56687: BlockingRecv error: recv got EOF from 127.0.98.67:56687 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 25 04:45:05 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:45:05 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:45:05.128120 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:45:05.122654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34911 (local address 127.0.98.65:37839)
0925 04:45:05.122929 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0925 04:45:05.122943 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:45:05.123077 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:45:05.123359 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:45:05.123361 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:45:05.123544 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:45:05.123643 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:45:05.123648 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:45:05.124409 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:45:05.124412 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:45:05.124971 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:45:05.124978 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:45:05.125399 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:45:05.125401 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:45:05.125460 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:05.125949 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:45:05.125960 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:45:05.128000 (+ 2040us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:34911: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":91,"threads_started":1}
W20260925 04:45:05.128356 9598 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:45:05.122724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:37839 (local address 127.0.98.126:34911)
0925 04:45:05.122970 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0925 04:45:05.122975 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:45:05.123033 (+ 58us) server_negotiation.cc:407] Connection header received
0925 04:45:05.123167 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:45:05.123169 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:45:05.123214 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:45:05.123295 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:45:05.123758 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:05.124304 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:05.125081 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:05.125303 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:05.125382 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:05.128175 (+ 2793us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.65:37839: BlockingRecv error: recv got EOF from 127.0.98.65:37839 (error 108)
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":64,"threads_started":1}
Sep 25 04:45:05 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:45:05 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:45:05.308926 9599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:45:05.303120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34911 (local address 127.0.98.66:60083)
0925 04:45:05.303439 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0925 04:45:05.303454 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:45:05.303601 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:45:05.303898 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:45:05.303901 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:45:05.304119 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:45:05.304216 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:45:05.304221 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:45:05.305013 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:45:05.305019 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:45:05.305608 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:45:05.305616 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:45:05.306156 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:45:05.306160 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:45:05.306303 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:05.306944 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:45:05.306957 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:45:05.308790 (+ 1833us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:34911: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":96,"threads_started":1}
W20260925 04:45:05.309052 9598 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:45:05.303201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:60083 (local address 127.0.98.126:34911)
0925 04:45:05.303381 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 04:45:05.303386 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:45:05.303548 (+ 162us) server_negotiation.cc:407] Connection header received
0925 04:45:05.303730 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:45:05.303733 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:45:05.303790 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:45:05.303879 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:45:05.304345 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:05.304901 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:05.305738 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:05.306025 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:05.306128 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:05.308933 (+ 2805us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.66:60083: BlockingRecv error: recv got EOF from 127.0.98.66:60083 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 25 04:45:05 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:45:05 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:45:05.513729 9600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:45:05.508106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34911 (local address 127.0.98.67:50915)
0925 04:45:05.508391 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0925 04:45:05.508407 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:45:05.508543 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:45:05.508821 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:45:05.508824 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:45:05.509035 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:45:05.509126 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:45:05.509139 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:45:05.509915 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:45:05.509920 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:45:05.510525 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:45:05.510533 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:45:05.510991 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:45:05.510995 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:45:05.511074 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:05.511655 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:45:05.511668 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:45:05.513596 (+ 1928us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:34911: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":113,"threads_started":1}
W20260925 04:45:05.513906 9598 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:45:05.508174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:50915 (local address 127.0.98.126:34911)
0925 04:45:05.508323 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 04:45:05.508327 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:45:05.508492 (+ 165us) server_negotiation.cc:407] Connection header received
0925 04:45:05.508653 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:45:05.508656 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:45:05.508711 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:45:05.508797 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:45:05.509251 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:05.509799 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:05.510652 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:05.510879 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:05.510991 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:05.513772 (+ 2781us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.67:50915: BlockingRecv error: recv got EOF from 127.0.98.67:50915 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 25 04:45:06 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:45:06 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:45:06.134405 9601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:45:06.128971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34911 (local address 127.0.98.65:42045)
0925 04:45:06.129241 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0925 04:45:06.129263 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 04:45:06.129355 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:45:06.129594 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:45:06.129596 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:45:06.129789 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:45:06.129883 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:45:06.129888 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:45:06.130690 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:45:06.130693 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:45:06.131263 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:45:06.131272 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:45:06.131696 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:45:06.131699 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:45:06.131760 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:06.132273 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:45:06.132285 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:45:06.134209 (+ 1924us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:34911: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":94,"threads_started":1}
W20260925 04:45:06.134526 9602 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:45:06.129038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:42045 (local address 127.0.98.126:34911)
0925 04:45:06.129300 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0925 04:45:06.129303 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:45:06.129317 (+ 14us) server_negotiation.cc:407] Connection header received
0925 04:45:06.129427 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:45:06.129430 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:45:06.129474 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:45:06.129554 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:45:06.129990 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:06.130575 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:06.131368 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:06.131588 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:06.131672 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:06.134393 (+ 2721us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.65:42045: BlockingRecv error: recv got EOF from 127.0.98.65:42045 (error 108)
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":117,"threads_started":1}
Sep 25 04:45:06 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:45:06 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:45:06.315688 9603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:45:06.309891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34911 (local address 127.0.98.66:44259)
0925 04:45:06.310301 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0925 04:45:06.310319 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 04:45:06.310486 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:45:06.310771 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:45:06.310774 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:45:06.310985 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:45:06.311079 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:45:06.311085 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:45:06.311858 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:45:06.311864 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:45:06.312425 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:45:06.312432 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:45:06.312898 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:45:06.312901 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:45:06.312988 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:06.313512 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:45:06.313524 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:45:06.315511 (+ 1987us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:34911: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":139,"threads_started":1}
W20260925 04:45:06.315752 9602 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:45:06.310037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:44259 (local address 127.0.98.126:34911)
0925 04:45:06.310300 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0925 04:45:06.310305 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:45:06.310412 (+ 107us) server_negotiation.cc:407] Connection header received
0925 04:45:06.310591 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:45:06.310594 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:45:06.310652 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:45:06.310747 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:45:06.311206 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:06.311742 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:06.312548 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:06.312785 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:06.312894 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:06.315641 (+ 2747us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.66:44259: BlockingRecv error: recv got EOF from 127.0.98.66:44259 (error 108)
Metrics: {"server-negotiator.queue_time_us":130}
Sep 25 04:45:06 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:45:06 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:45:06.523905 9604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:45:06.514706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34911 (local address 127.0.98.67:36675)
0925 04:45:06.514976 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0925 04:45:06.514989 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:45:06.515103 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:45:06.515388 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:45:06.515391 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:45:06.515578 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:45:06.515676 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:45:06.515681 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:45:06.516409 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:45:06.516412 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:45:06.516979 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:45:06.516987 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:45:06.517416 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:45:06.517419 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:45:06.517483 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:06.521752 (+ 4269us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:45:06.521765 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:45:06.523786 (+ 2021us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:34911: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":91,"threads_started":1}
W20260925 04:45:06.524055 9602 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:45:06.514782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:36675 (local address 127.0.98.126:34911)
0925 04:45:06.514941 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 04:45:06.514945 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:45:06.515067 (+ 122us) server_negotiation.cc:407] Connection header received
0925 04:45:06.515188 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:45:06.515190 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:45:06.515238 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:45:06.515317 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:45:06.515783 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:06.516298 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:06.517110 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:06.517325 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:06.517398 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:06.523935 (+ 6537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.67:36675: BlockingRecv error: recv got EOF from 127.0.98.67:36675 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 25 04:45:07 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:45:07 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:45:07.141657 9605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:45:07.135480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34911 (local address 127.0.98.65:52319)
0925 04:45:07.135758 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0925 04:45:07.135773 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:45:07.135873 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:45:07.136160 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:45:07.136162 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:45:07.136349 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:45:07.136443 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:45:07.136448 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:45:07.137496 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:45:07.137499 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:45:07.138198 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:45:07.138209 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:45:07.138863 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:45:07.138867 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:45:07.138977 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:07.139582 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:45:07.139594 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:45:07.141536 (+ 1942us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:34911: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":91,"threads_started":1}
W20260925 04:45:07.141868 9606 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:45:07.135559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:52319 (local address 127.0.98.126:34911)
0925 04:45:07.135817 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0925 04:45:07.135821 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:45:07.135837 (+ 16us) server_negotiation.cc:407] Connection header received
0925 04:45:07.135972 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:45:07.135975 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:45:07.136021 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:45:07.136105 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:45:07.136726 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:07.137373 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:07.138393 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:07.138715 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:07.138864 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:07.141713 (+ 2849us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.65:52319: BlockingRecv error: recv got EOF from 127.0.98.65:52319 (error 108)
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":74,"threads_started":1}
Sep 25 04:45:07 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:45:07 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:45:07.322839 9607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:45:07.316754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34911 (local address 127.0.98.66:37911)
0925 04:45:07.317212 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0925 04:45:07.317229 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 04:45:07.317348 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:45:07.317660 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:45:07.317663 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:45:07.317908 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:45:07.318033 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:45:07.318086 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:45:07.318857 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:45:07.318860 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:45:07.319444 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:45:07.319451 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:45:07.319940 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:45:07.319943 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:45:07.320011 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:07.320579 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:45:07.320591 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:45:07.322708 (+ 2117us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:34911: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":101,"threads_started":1}
W20260925 04:45:07.323053 9606 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:45:07.316921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:37911 (local address 127.0.98.126:34911)
0925 04:45:07.317089 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 04:45:07.317094 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:45:07.317313 (+ 219us) server_negotiation.cc:407] Connection header received
0925 04:45:07.317455 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:45:07.317458 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:45:07.317521 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:45:07.317613 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:45:07.318214 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:07.318761 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:07.319577 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:07.319800 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:07.319891 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:07.322887 (+ 2996us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.66:37911: BlockingRecv error: recv got EOF from 127.0.98.66:37911 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 25 04:45:07 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:45:07 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:45:07.529997 9608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:45:07.524734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34911 (local address 127.0.98.67:45025)
0925 04:45:07.525010 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0925 04:45:07.525027 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 04:45:07.525131 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:45:07.525379 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:45:07.525381 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:45:07.525556 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:45:07.525671 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:45:07.525676 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:45:07.526458 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:45:07.526461 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:45:07.527062 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:45:07.527072 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:45:07.527484 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:45:07.527487 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:45:07.527561 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:07.528046 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:45:07.528057 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:45:07.529825 (+ 1768us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:34911: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":100,"threads_started":1}
W20260925 04:45:07.530068 9606 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:45:07.524881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:45025 (local address 127.0.98.126:34911)
0925 04:45:07.525016 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 04:45:07.525021 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:45:07.525100 (+ 79us) server_negotiation.cc:407] Connection header received
0925 04:45:07.525219 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:45:07.525222 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:45:07.525267 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:45:07.525343 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:45:07.525772 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:07.526336 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:07.527189 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:07.527395 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:07.527469 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:07.529957 (+ 2488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.67:45025: BlockingRecv error: recv got EOF from 127.0.98.67:45025 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 25 04:45:08 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:45:08 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:45:08.148010 9609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:45:08.142571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34911 (local address 127.0.98.65:33523)
0925 04:45:08.142879 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0925 04:45:08.142900 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 04:45:08.143024 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:45:08.143283 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:45:08.143285 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:45:08.143475 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:45:08.143571 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:45:08.143576 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:45:08.144351 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:45:08.144354 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:45:08.144915 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:45:08.144922 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:45:08.145354 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:45:08.145356 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:45:08.145417 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:08.145896 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:45:08.145908 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:45:08.147825 (+ 1917us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:34911: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":88,"threads_started":1}
W20260925 04:45:08.148259 9610 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:45:08.142663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:33523 (local address 127.0.98.126:34911)
0925 04:45:08.142992 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0925 04:45:08.142996 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:45:08.143010 (+ 14us) server_negotiation.cc:407] Connection header received
0925 04:45:08.143123 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:45:08.143125 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:45:08.143168 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:45:08.143245 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:45:08.143686 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:08.144255 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:08.145011 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:08.145256 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:08.145332 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:08.147949 (+ 2617us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.65:33523: BlockingRecv error: recv got EOF from 127.0.98.65:33523 (error 108)
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":73,"threads_started":1}
Sep 25 04:45:08 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:45:08 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:45:08.329370 9611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:45:08.323674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34911 (local address 127.0.98.66:33141)
0925 04:45:08.324009 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0925 04:45:08.324027 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 04:45:08.324163 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:45:08.324424 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:45:08.324427 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:45:08.324644 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:45:08.324735 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:45:08.324741 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:45:08.325515 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:45:08.325521 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:45:08.326101 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:45:08.326108 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:45:08.326629 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:45:08.326633 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:45:08.326737 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:08.327320 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:45:08.327333 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:45:08.329176 (+ 1843us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:34911: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":105,"threads_started":1}
W20260925 04:45:08.329409 9610 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:45:08.323810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:33141 (local address 127.0.98.126:34911)
0925 04:45:08.323966 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 04:45:08.323971 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:45:08.324105 (+ 134us) server_negotiation.cc:407] Connection header received
0925 04:45:08.324261 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:45:08.324265 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:45:08.324315 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:45:08.324406 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:45:08.324860 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:08.325402 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:08.326219 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:08.326506 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:08.326627 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:08.329300 (+ 2673us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.66:33141: BlockingRecv error: recv got EOF from 127.0.98.66:33141 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 25 04:45:08 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:45:08 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:45:08.536958 9612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:45:08.530908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34911 (local address 127.0.98.67:59551)
0925 04:45:08.531309 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0925 04:45:08.531323 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:45:08.531471 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:45:08.531781 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:45:08.531784 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:45:08.531997 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:45:08.532085 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:45:08.532091 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:45:08.532912 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:45:08.532918 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:45:08.533491 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:45:08.533499 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:45:08.533975 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:45:08.533979 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:45:08.534061 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:08.534705 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:45:08.534718 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:45:08.536831 (+ 2113us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:34911: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":144,"threads_started":1}
W20260925 04:45:08.537200 9610 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:45:08.531041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:59551 (local address 127.0.98.126:34911)
0925 04:45:08.531224 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 04:45:08.531228 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:45:08.531407 (+ 179us) server_negotiation.cc:407] Connection header received
0925 04:45:08.531593 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:45:08.531601 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:45:08.531666 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:45:08.531756 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:45:08.532256 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:08.532796 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:08.533619 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:08.533856 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:08.533969 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:08.537035 (+ 3066us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.67:59551: BlockingRecv error: recv got EOF from 127.0.98.67:59551 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 25 04:45:09 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:45:09 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:45:09.154727 9613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:45:09.148914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34911 (local address 127.0.98.65:36005)
0925 04:45:09.149227 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0925 04:45:09.149241 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:45:09.149341 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:45:09.149602 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:45:09.149604 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:45:09.149803 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:45:09.149897 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:45:09.149902 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:45:09.150706 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:45:09.150709 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:45:09.151282 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:45:09.151291 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:45:09.151714 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:45:09.151716 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:45:09.151777 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:09.152352 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:45:09.152364 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:45:09.154591 (+ 2227us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:34911: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":84,"threads_started":1}
W20260925 04:45:09.154958 9614 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:45:09.149050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:36005 (local address 127.0.98.126:34911)
0925 04:45:09.149299 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0925 04:45:09.149303 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:45:09.149317 (+ 14us) server_negotiation.cc:407] Connection header received
0925 04:45:09.149438 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:45:09.149440 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:45:09.149483 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:45:09.149563 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:45:09.149997 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:09.150592 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:09.151381 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:09.151611 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:09.151686 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:09.154809 (+ 3123us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.65:36005: BlockingRecv error: recv got EOF from 127.0.98.65:36005 (error 108)
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":56,"threads_started":1}
Sep 25 04:45:09 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:45:09 dist-test-slave-kghz krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:45:09.336606 9615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:45:09.330527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34911 (local address 127.0.98.66:52677)
0925 04:45:09.330807 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0925 04:45:09.330821 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:45:09.330917 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:45:09.331189 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:45:09.331191 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:45:09.331375 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:45:09.331469 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:45:09.331474 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:45:09.332649 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:45:09.332655 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:45:09.333406 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:45:09.333418 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:45:09.333870 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:45:09.333873 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:45:09.333955 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:09.334576 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:45:09.334597 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:45:09.336418 (+ 1821us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:34911: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":90,"threads_started":1}
W20260925 04:45:09.336663 9614 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:45:09.330618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:52677 (local address 127.0.98.126:34911)
0925 04:45:09.330783 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 04:45:09.330787 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:45:09.330890 (+ 103us) server_negotiation.cc:407] Connection header received
0925 04:45:09.330998 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:45:09.331000 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:45:09.331077 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:45:09.331152 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:45:09.331651 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:09.332481 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:09.333538 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:09.333759 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:09.333864 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:09.336544 (+ 2680us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.66:52677: BlockingRecv error: recv got EOF from 127.0.98.66:52677 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 04:45:09.415261 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 9139
I20260925 04:45:09.421195 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 9291
I20260925 04:45:09.426781 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 9424
I20260925 04:45:09.432488 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 9069
2026-09-25T04:45:09Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18076 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-25T04:45:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:45:09Z Disabled control of system clock
I20260925 04:45:09.465124 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311449873301-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311449873301-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311449873301-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:35931
--webserver_interface=127.0.98.126
--webserver_port=0
--builtin_ntp_servers=127.0.98.84:36637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:35931
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 04:45:09.579578 9619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:45:09.579866 9619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:45:09.579901 9619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:45:09.584278 9619 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:45:09.584350 9619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:45:09.584367 9619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:45:09.584384 9619 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:45:09.584399 9619 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:45:09.589274 9619 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:36637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311449873301-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311449873301-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:35931
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:35931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9619
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311449873301-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:45:09.590507 9619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:45:09.591954 9619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:45:09.601219 9624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:45:09.601204 9625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:45:09.601384 9619 server_base.cc:1034] running on GCE node
W20260925 04:45:09.601187 9627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:45:09.602002 9619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:45:09.603322 9619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:45:09.604600 9619 hybrid_clock.cc:648] HybridClock initialized: now 1790311509604586 us; error 31 us; skew 500 ppm
I20260925 04:45:09.606796 9619 webserver.cc:460] Webserver started at http://127.0.98.126:44599/ using document root <none> and password file <none>
I20260925 04:45:09.607462 9619 fs_manager.cc:359] Metadata directory not provided
I20260925 04:45:09.607534 9619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:45:09.607791 9619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:45:09.609979 9619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311449873301-393-0/minicluster-data/master-0/data/instance:
uuid: "d8baea461cdf492bad64c0672f7aba35"
format_stamp: "Formatted at 2026-09-25 04:45:09 on dist-test-slave-kghz"
server_key: "00c4c5791ac9e936bd3047c190f2650e"
server_key_iv: "c6c06f1505ea073fb109097efb655835"
server_key_version: "encryptionkey@0"
I20260925 04:45:09.610634 9619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311449873301-393-0/minicluster-data/master-0/wal/instance:
uuid: "d8baea461cdf492bad64c0672f7aba35"
format_stamp: "Formatted at 2026-09-25 04:45:09 on dist-test-slave-kghz"
server_key: "00c4c5791ac9e936bd3047c190f2650e"
server_key_iv: "c6c06f1505ea073fb109097efb655835"
server_key_version: "encryptionkey@0"
I20260925 04:45:09.614630 9619 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 04:45:09.617408 9633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:45:09.618826 9619 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:45:09.619019 9619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311449873301-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311449873301-393-0/minicluster-data/master-0/wal
uuid: "d8baea461cdf492bad64c0672f7aba35"
format_stamp: "Formatted at 2026-09-25 04:45:09 on dist-test-slave-kghz"
server_key: "00c4c5791ac9e936bd3047c190f2650e"
server_key_iv: "c6c06f1505ea073fb109097efb655835"
server_key_version: "encryptionkey@0"
I20260925 04:45:09.619163 9619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311449873301-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311449873301-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311449873301-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:45:09.633217 9619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:45:09.633996 9619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:45:09.634220 9619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:45:09.641630 9619 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:35931
I20260925 04:45:09.641654 9685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:35931 every 8 connection(s)
I20260925 04:45:09.642724 9619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
I20260925 04:45:09.645972 9686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:45:09.651544 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 9619
I20260925 04:45:09.651687 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311449873301-393-0/minicluster-data/master-0/wal/instance
I20260925 04:45:09.651806 9686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8baea461cdf492bad64c0672f7aba35: Bootstrap starting.
I20260925 04:45:09.652016 393 external_mini_cluster.cc:1428] Setting key 2aeeef5330e3c31c971a6debbad84f24
I20260925 04:45:09.654592 9686 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8baea461cdf492bad64c0672f7aba35: Neither blocks nor log segments found. Creating new log.
I20260925 04:45:09.655362 9686 log.cc:824] T 00000000000000000000000000000000 P d8baea461cdf492bad64c0672f7aba35: Log is configured to *not* fsync() on all Append() calls
I20260925 04:45:09.657305 9686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8baea461cdf492bad64c0672f7aba35: No bootstrap required, opened a new log
I20260925 04:45:09.661404 9686 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8baea461cdf492bad64c0672f7aba35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8baea461cdf492bad64c0672f7aba35" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 35931 } }
I20260925 04:45:09.661636 9686 raft_consensus.cc:374] T 00000000000000000000000000000000 P d8baea461cdf492bad64c0672f7aba35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:45:09.661693 9686 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8baea461cdf492bad64c0672f7aba35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8baea461cdf492bad64c0672f7aba35, State: Initialized, Role: FOLLOWER
I20260925 04:45:09.662166 9689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:45:09.653413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44760 (local address 127.0.98.126:35931)
0925 04:45:09.654126 (+ 713us) server_negotiation.cc:206] Beginning negotiation
0925 04:45:09.654141 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 04:45:09.654178 (+ 37us) server_negotiation.cc:407] Connection header received
0925 04:45:09.654969 (+ 791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:45:09.655032 (+ 63us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:45:09.655486 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:45:09.655848 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:45:09.656290 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:09.657271 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:09.658375 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:09.658875 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:09.659049 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:09.660143 (+ 1094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:45:09.660149 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:45:09.660164 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:45:09.660671 (+ 507us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:45:09.660730 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:45:09.660751 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:45:09.660869 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 04:45:09.661001 (+ 132us) server_negotiation.cc:299] Negotiation successful
0925 04:45:09.661271 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":149,"threads_started":1}
I20260925 04:45:09.662137 9686 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8baea461cdf492bad64c0672f7aba35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8baea461cdf492bad64c0672f7aba35" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 35931 } }
I20260925 04:45:09.662346 9686 raft_consensus.cc:388] T 00000000000000000000000000000000 P d8baea461cdf492bad64c0672f7aba35 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:45:09.662443 9686 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8baea461cdf492bad64c0672f7aba35 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:45:09.662554 9686 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8baea461cdf492bad64c0672f7aba35 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:45:09.663534 9686 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8baea461cdf492bad64c0672f7aba35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8baea461cdf492bad64c0672f7aba35" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 35931 } }
I20260925 04:45:09.663872 9686 leader_election.cc:302] T 00000000000000000000000000000000 P d8baea461cdf492bad64c0672f7aba35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8baea461cdf492bad64c0672f7aba35; no voters:
I20260925 04:45:09.664216 9686 leader_election.cc:288] T 00000000000000000000000000000000 P d8baea461cdf492bad64c0672f7aba35 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:45:09.664286 9691 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8baea461cdf492bad64c0672f7aba35 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:45:09.664485 9691 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8baea461cdf492bad64c0672f7aba35 [term 1 LEADER]: Becoming Leader. State: Replica: d8baea461cdf492bad64c0672f7aba35, State: Running, Role: LEADER
I20260925 04:45:09.664841 9691 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8baea461cdf492bad64c0672f7aba35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8baea461cdf492bad64c0672f7aba35" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 35931 } }
I20260925 04:45:09.665506 9686 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8baea461cdf492bad64c0672f7aba35 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:45:09.666656 9692 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8baea461cdf492bad64c0672f7aba35 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8baea461cdf492bad64c0672f7aba35. Latest consensus state: current_term: 1 leader_uuid: "d8baea461cdf492bad64c0672f7aba35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8baea461cdf492bad64c0672f7aba35" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 35931 } } }
I20260925 04:45:09.666678 9693 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8baea461cdf492bad64c0672f7aba35 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d8baea461cdf492bad64c0672f7aba35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8baea461cdf492bad64c0672f7aba35" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 35931 } } }
I20260925 04:45:09.666774 9692 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8baea461cdf492bad64c0672f7aba35 [sys.catalog]: This master's current role is: LEADER
I20260925 04:45:09.666785 9693 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8baea461cdf492bad64c0672f7aba35 [sys.catalog]: This master's current role is: LEADER
I20260925 04:45:09.667201 9700 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:45:09.670047 9700 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:45:09.675837 9700 catalog_manager.cc:1347] Generated new cluster ID: 514ee3b085db4c7aa9f266f0145c9f97
I20260925 04:45:09.675922 9700 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:45:09.695369 9700 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:45:09.696411 9700 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:45:09.703207 9700 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d8baea461cdf492bad64c0672f7aba35: Generated new TSK 0
I20260925 04:45:09.703999 9700 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:45:09.708870 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311449873301-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311449873301-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311449873301-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:0
--local_ip_for_outbound_sockets=127.0.98.65
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:35931
--builtin_ntp_servers=127.0.98.84:36637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260925 04:45:09.826735 9710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:45:09.827020 9710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:45:09.827067 9710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:45:09.831199 9710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:45:09.831264 9710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:45:09.831351 9710 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260925 04:45:09.836440 9710 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:36637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311449873301-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311449873301-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:35931
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9710
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311449873301-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:45:09.837626 9710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:45:09.839151 9710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:45:09.848852 9718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:45:09.848877 9716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:45:09.848867 9715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:45:09.849262 9710 server_base.cc:1034] running on GCE node
I20260925 04:45:09.849656 9710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:45:09.850332 9710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:45:09.851518 9710 hybrid_clock.cc:648] HybridClock initialized: now 1790311509851499 us; error 29 us; skew 500 ppm
I20260925 04:45:09.853503 9710 webserver.cc:460] Webserver started at http://127.0.98.65:33689/ using document root <none> and password file <none>
I20260925 04:45:09.854127 9710 fs_manager.cc:359] Metadata directory not provided
I20260925 04:45:09.854198 9710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:45:09.854431 9710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:45:09.857416 9710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311449873301-393-0/minicluster-data/ts-0/data/instance:
uuid: "f056474caca54ff394109dc1f6978261"
format_stamp: "Formatted at 2026-09-25 04:45:09 on dist-test-slave-kghz"
server_key: "5a9b36f2cd32b4ed49d93543c222cf3b"
server_key_iv: "dfb9eb8e1b947ea67ec4e01e958f6b8e"
server_key_version: "encryptionkey@0"
I20260925 04:45:09.858078 9710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311449873301-393-0/minicluster-data/ts-0/wal/instance:
uuid: "f056474caca54ff394109dc1f6978261"
format_stamp: "Formatted at 2026-09-25 04:45:09 on dist-test-slave-kghz"
server_key: "5a9b36f2cd32b4ed49d93543c222cf3b"
server_key_iv: "dfb9eb8e1b947ea67ec4e01e958f6b8e"
server_key_version: "encryptionkey@0"
I20260925 04:45:09.861899 9710 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260925 04:45:09.864629 9724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:45:09.866058 9710 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 04:45:09.866266 9710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311449873301-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311449873301-393-0/minicluster-data/ts-0/wal
uuid: "f056474caca54ff394109dc1f6978261"
format_stamp: "Formatted at 2026-09-25 04:45:09 on dist-test-slave-kghz"
server_key: "5a9b36f2cd32b4ed49d93543c222cf3b"
server_key_iv: "dfb9eb8e1b947ea67ec4e01e958f6b8e"
server_key_version: "encryptionkey@0"
I20260925 04:45:09.866405 9710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311449873301-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311449873301-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311449873301-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:45:09.896945 9710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:45:09.897728 9710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:45:09.897933 9710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:45:09.898610 9710 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:45:09.899586 9710 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:45:09.899665 9710 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:45:09.899732 9710 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:45:09.899776 9710 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:45:09.908818 9710 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:35349
I20260925 04:45:09.908840 9837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:35349 every 8 connection(s)
I20260925 04:45:09.909773 9710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
I20260925 04:45:09.916021 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 9710
I20260925 04:45:09.916147 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311449873301-393-0/minicluster-data/ts-0/wal/instance
I20260925 04:45:09.916433 393 external_mini_cluster.cc:1428] Setting key 70b11cd8e7189ec763f31f69e808e511
I20260925 04:45:09.918675 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311449873301-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311449873301-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311449873301-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:0
--local_ip_for_outbound_sockets=127.0.98.66
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:35931
--builtin_ntp_servers=127.0.98.84:36637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 04:45:09.920192 9689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:45:09.911895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:38475 (local address 127.0.98.126:35931)
0925 04:45:09.912039 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 04:45:09.912043 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:45:09.913147 (+ 1104us) server_negotiation.cc:407] Connection header received
0925 04:45:09.914642 (+ 1495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:45:09.914646 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:45:09.914714 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:45:09.914792 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:45:09.915777 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:09.916401 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:09.917978 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:09.918306 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:09.918415 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:09.919618 (+ 1203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:45:09.919625 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:45:09.919632 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:45:09.919651 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:45:09.919679 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:45:09.919692 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:45:09.919778 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 04:45:09.919992 (+ 214us) server_negotiation.cc:299] Negotiation successful
0925 04:45:09.920057 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 04:45:09.921123 9840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:45:09.912317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:35931 (local address 127.0.98.65:38475)
0925 04:45:09.912988 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0925 04:45:09.913030 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 04:45:09.914421 (+ 1391us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:45:09.914963 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:45:09.914971 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:45:09.915057 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:45:09.915627 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:45:09.915640 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:45:09.916532 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:45:09.916537 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:45:09.917821 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:45:09.917834 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:45:09.918447 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:45:09.918450 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:45:09.918588 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:09.919307 (+ 719us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:45:09.919309 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:45:09.919351 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:45:09.919435 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:45:09.919810 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:45:09.919819 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:45:09.919825 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 04:45:09.919991 (+ 166us) client_negotiation.cc:240] Negotiation successful
0925 04:45:09.920298 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":111,"threads_started":1}
I20260925 04:45:09.922534 9838 heartbeater.cc:347] Connected to a master server at 127.0.98.126:35931
I20260925 04:45:09.922804 9838 heartbeater.cc:464] Registering TS with master...
I20260925 04:45:09.923365 9838 heartbeater.cc:511] Master 127.0.98.126:35931 requested a full tablet report, sending...
I20260925 04:45:09.924847 9643 ts_manager.cc:194] Registered new tserver with Master: f056474caca54ff394109dc1f6978261 (127.0.98.65:35349)
I20260925 04:45:09.926000 9643 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.65:38475
W20260925 04:45:10.036319 9841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:45:10.036648 9841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:45:10.036722 9841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:45:10.040952 9841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:45:10.041070 9841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:45:10.041167 9841 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260925 04:45:10.046844 9841 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:36637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311449873301-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311449873301-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:35931
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311449873301-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:45:10.048003 9841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:45:10.049474 9841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:45:10.059883 9849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:45:10.059963 9841 server_base.cc:1034] running on GCE node
W20260925 04:45:10.059966 9846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:45:10.059973 9847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:45:10.060642 9841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:45:10.061306 9841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:45:10.062484 9841 hybrid_clock.cc:648] HybridClock initialized: now 1790311510062451 us; error 61 us; skew 500 ppm
I20260925 04:45:10.064471 9841 webserver.cc:460] Webserver started at http://127.0.98.66:44915/ using document root <none> and password file <none>
I20260925 04:45:10.065065 9841 fs_manager.cc:359] Metadata directory not provided
I20260925 04:45:10.065128 9841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:45:10.065368 9841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:45:10.067989 9841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311449873301-393-0/minicluster-data/ts-1/data/instance:
uuid: "8b71ec36b7124130b23549a9e7031893"
format_stamp: "Formatted at 2026-09-25 04:45:10 on dist-test-slave-kghz"
server_key: "0114345090cb55c9b296c4db7f2cf691"
server_key_iv: "70b6b847355fa37a48294a411556787a"
server_key_version: "encryptionkey@0"
I20260925 04:45:10.068534 9841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311449873301-393-0/minicluster-data/ts-1/wal/instance:
uuid: "8b71ec36b7124130b23549a9e7031893"
format_stamp: "Formatted at 2026-09-25 04:45:10 on dist-test-slave-kghz"
server_key: "0114345090cb55c9b296c4db7f2cf691"
server_key_iv: "70b6b847355fa37a48294a411556787a"
server_key_version: "encryptionkey@0"
I20260925 04:45:10.072592 9841 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 04:45:10.075430 9855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:45:10.076749 9841 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:45:10.076856 9841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311449873301-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311449873301-393-0/minicluster-data/ts-1/wal
uuid: "8b71ec36b7124130b23549a9e7031893"
format_stamp: "Formatted at 2026-09-25 04:45:10 on dist-test-slave-kghz"
server_key: "0114345090cb55c9b296c4db7f2cf691"
server_key_iv: "70b6b847355fa37a48294a411556787a"
server_key_version: "encryptionkey@0"
I20260925 04:45:10.076957 9841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311449873301-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311449873301-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311449873301-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:45:10.087318 9841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:45:10.088021 9841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:45:10.088198 9841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:45:10.088768 9841 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:45:10.089725 9841 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:45:10.089771 9841 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:45:10.089809 9841 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:45:10.089825 9841 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:45:10.099576 9841 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:35571
I20260925 04:45:10.099596 9968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:35571 every 8 connection(s)
I20260925 04:45:10.100566 9841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
I20260925 04:45:10.105865 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 9841
I20260925 04:45:10.106017 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311449873301-393-0/minicluster-data/ts-1/wal/instance
I20260925 04:45:10.106395 393 external_mini_cluster.cc:1428] Setting key 2b3e1e7abae17fe398bceef15506dcbb
I20260925 04:45:10.108445 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311449873301-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311449873301-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311449873301-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:0
--local_ip_for_outbound_sockets=127.0.98.67
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:35931
--builtin_ntp_servers=127.0.98.84:36637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 04:45:10.110991 9689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:45:10.102600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:47565 (local address 127.0.98.126:35931)
0925 04:45:10.102736 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 04:45:10.102740 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:45:10.103922 (+ 1182us) server_negotiation.cc:407] Connection header received
0925 04:45:10.105425 (+ 1503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:45:10.105428 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:45:10.105490 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:45:10.105605 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:45:10.106769 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:10.107410 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:10.108733 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:10.108956 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:10.109083 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:10.110458 (+ 1375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:45:10.110461 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:45:10.110464 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:45:10.110477 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:45:10.110497 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:45:10.110505 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:45:10.110589 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 04:45:10.110807 (+ 218us) server_negotiation.cc:299] Negotiation successful
0925 04:45:10.110866 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 04:45:10.112308 9971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:45:10.103010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:35931 (local address 127.0.98.66:47565)
0925 04:45:10.103748 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0925 04:45:10.103790 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 04:45:10.105195 (+ 1405us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:45:10.105810 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:45:10.105818 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:45:10.105906 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:45:10.106616 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:45:10.106632 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:45:10.107538 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:45:10.107541 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:45:10.108549 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:45:10.108589 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:45:10.109116 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:45:10.109119 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:45:10.109303 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:10.110060 (+ 757us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:45:10.110063 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:45:10.110135 (+ 72us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:45:10.110235 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:45:10.110616 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:45:10.110629 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:45:10.110635 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 04:45:10.110781 (+ 146us) client_negotiation.cc:240] Negotiation successful
0925 04:45:10.111184 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":127,"threads_started":1}
I20260925 04:45:10.113777 9969 heartbeater.cc:347] Connected to a master server at 127.0.98.126:35931
I20260925 04:45:10.114055 9969 heartbeater.cc:464] Registering TS with master...
I20260925 04:45:10.114681 9969 heartbeater.cc:511] Master 127.0.98.126:35931 requested a full tablet report, sending...
I20260925 04:45:10.115890 9643 ts_manager.cc:194] Registered new tserver with Master: 8b71ec36b7124130b23549a9e7031893 (127.0.98.66:35571)
I20260925 04:45:10.116573 9643 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.66:47565
W20260925 04:45:10.231781 9972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:45:10.232105 9972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:45:10.232177 9972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:45:10.236234 9972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:45:10.236317 9972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:45:10.236406 9972 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260925 04:45:10.241418 9972 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:36637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311449873301-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311449873301-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:35931
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9972
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311449873301-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:45:10.242586 9972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:45:10.243963 9972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:45:10.254033 9978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:45:10.254211 9977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:45:10.254083 9980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:45:10.254439 9972 server_base.cc:1034] running on GCE node
I20260925 04:45:10.254858 9972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:45:10.255636 9972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:45:10.256822 9972 hybrid_clock.cc:648] HybridClock initialized: now 1790311510256784 us; error 50 us; skew 500 ppm
I20260925 04:45:10.258802 9972 webserver.cc:460] Webserver started at http://127.0.98.67:45563/ using document root <none> and password file <none>
I20260925 04:45:10.259423 9972 fs_manager.cc:359] Metadata directory not provided
I20260925 04:45:10.259488 9972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:45:10.259660 9972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:45:10.261716 9972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311449873301-393-0/minicluster-data/ts-2/data/instance:
uuid: "82d81455ccf4477ba10deb95c5151a07"
format_stamp: "Formatted at 2026-09-25 04:45:10 on dist-test-slave-kghz"
server_key: "fc390d4e0da86448e4c05d0237299680"
server_key_iv: "3eb562de23304ac7c0acd392d99fd5be"
server_key_version: "encryptionkey@0"
I20260925 04:45:10.262411 9972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311449873301-393-0/minicluster-data/ts-2/wal/instance:
uuid: "82d81455ccf4477ba10deb95c5151a07"
format_stamp: "Formatted at 2026-09-25 04:45:10 on dist-test-slave-kghz"
server_key: "fc390d4e0da86448e4c05d0237299680"
server_key_iv: "3eb562de23304ac7c0acd392d99fd5be"
server_key_version: "encryptionkey@0"
I20260925 04:45:10.266371 9972 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 04:45:10.269174 9986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:45:10.270642 9972 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:45:10.270758 9972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311449873301-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311449873301-393-0/minicluster-data/ts-2/wal
uuid: "82d81455ccf4477ba10deb95c5151a07"
format_stamp: "Formatted at 2026-09-25 04:45:10 on dist-test-slave-kghz"
server_key: "fc390d4e0da86448e4c05d0237299680"
server_key_iv: "3eb562de23304ac7c0acd392d99fd5be"
server_key_version: "encryptionkey@0"
I20260925 04:45:10.270975 9972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311449873301-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311449873301-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311449873301-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:45:10.287850 9972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:45:10.288590 9972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:45:10.288796 9972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:45:10.289408 9972 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:45:10.290544 9972 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:45:10.290606 9972 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:45:10.290647 9972 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:45:10.290705 9972 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:45:10.299927 9972 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:44803
I20260925 04:45:10.299944 10099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:44803 every 8 connection(s)
I20260925 04:45:10.300938 9972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
I20260925 04:45:10.306053 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 9972
I20260925 04:45:10.306143 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790311449873301-393-0/minicluster-data/ts-2/wal/instance
I20260925 04:45:10.306463 393 external_mini_cluster.cc:1428] Setting key d613276427824e62ceea77281d03bcaa
I20260925 04:45:10.310904 9689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:45:10.303127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:53751 (local address 127.0.98.126:35931)
0925 04:45:10.303295 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 04:45:10.303299 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:45:10.304385 (+ 1086us) server_negotiation.cc:407] Connection header received
0925 04:45:10.305809 (+ 1424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:45:10.305812 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:45:10.305873 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:45:10.305987 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:45:10.307180 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:10.307731 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:10.308884 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:10.309120 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:10.309231 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:10.310388 (+ 1157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:45:10.310396 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:45:10.310398 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:45:10.310409 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:45:10.310428 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:45:10.310436 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:45:10.310542 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 04:45:10.310695 (+ 153us) server_negotiation.cc:299] Negotiation successful
0925 04:45:10.310758 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 04:45:10.311811 10102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:45:10.303516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:35931 (local address 127.0.98.67:53751)
0925 04:45:10.304215 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0925 04:45:10.304257 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 04:45:10.305588 (+ 1331us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:45:10.306215 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:45:10.306223 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:45:10.306369 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:45:10.307031 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:45:10.307048 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:45:10.307866 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:45:10.307869 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:45:10.308756 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:45:10.308764 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:45:10.309271 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:45:10.309275 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:45:10.309445 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:10.310092 (+ 647us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:45:10.310093 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:45:10.310135 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:45:10.310200 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:45:10.310520 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:45:10.310529 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:45:10.310535 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 04:45:10.310695 (+ 160us) client_negotiation.cc:240] Negotiation successful
0925 04:45:10.310988 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":114,"threads_started":1}
I20260925 04:45:10.313045 10100 heartbeater.cc:347] Connected to a master server at 127.0.98.126:35931
I20260925 04:45:10.313321 10100 heartbeater.cc:464] Registering TS with master...
I20260925 04:45:10.313870 10100 heartbeater.cc:511] Master 127.0.98.126:35931 requested a full tablet report, sending...
I20260925 04:45:10.315100 9643 ts_manager.cc:194] Registered new tserver with Master: 82d81455ccf4477ba10deb95c5151a07 (127.0.98.67:44803)
I20260925 04:45:10.315644 9643 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.67:53751
I20260925 04:45:10.320312 393 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 04:45:10.323298 9689 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:45:10.322190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44770 (local address 127.0.98.126:35931)
0925 04:45:10.322351 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 04:45:10.322356 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:45:10.322533 (+ 177us) server_negotiation.cc:407] Connection header received
0925 04:45:10.322707 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:45:10.322710 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:45:10.322762 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:45:10.322848 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:45:10.323134 (+ 286us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44770: BlockingRecv error: recv got EOF from 127.0.0.1:44770 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 04:45:10.323524 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 9710
I20260925 04:45:10.329600 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 9841
I20260925 04:45:10.335372 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 9972
I20260925 04:45:10.341056 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 9619
2026-09-25T04:45:10Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (906 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-25T04:45:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:45:10Z Disabled control of system clock
I20260925 04:45:10.371011 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311449873301-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311449873301-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311449873301-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:37607
--webserver_interface=127.0.98.126
--webserver_port=0
--builtin_ntp_servers=127.0.98.84:41303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:37607
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 04:45:10.484638 10111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:45:10.484931 10111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:45:10.484970 10111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:45:10.488981 10111 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:45:10.489058 10111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:45:10.489074 10111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:45:10.489091 10111 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:45:10.489106 10111 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:45:10.493932 10111 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:41303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311449873301-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311449873301-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:37607
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:37607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311449873301-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:45:10.495150 10111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:45:10.496549 10111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 04:45:10.505287 10111 server_base.cc:1034] running on GCE node
W20260925 04:45:10.505334 10117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:45:10.505406 10116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:45:10.505246 10119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:45:10.506114 10111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:45:10.507321 10111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:45:10.508498 10111 hybrid_clock.cc:648] HybridClock initialized: now 1790311510508461 us; error 50 us; skew 500 ppm
I20260925 04:45:10.510529 10111 webserver.cc:460] Webserver started at http://127.0.98.126:42525/ using document root <none> and password file <none>
I20260925 04:45:10.511138 10111 fs_manager.cc:359] Metadata directory not provided
I20260925 04:45:10.511202 10111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:45:10.511446 10111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:45:10.513567 10111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311449873301-393-0/minicluster-data/master-0/data/instance:
uuid: "49cbe4adf79c47ec931a003dbf520e3d"
format_stamp: "Formatted at 2026-09-25 04:45:10 on dist-test-slave-kghz"
server_key: "c01b4dff4cda0c8e8b9351e8fe79e844"
server_key_iv: "0c134f68477f2c65d311f57fb450b957"
server_key_version: "encryptionkey@0"
I20260925 04:45:10.514138 10111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311449873301-393-0/minicluster-data/master-0/wal/instance:
uuid: "49cbe4adf79c47ec931a003dbf520e3d"
format_stamp: "Formatted at 2026-09-25 04:45:10 on dist-test-slave-kghz"
server_key: "c01b4dff4cda0c8e8b9351e8fe79e844"
server_key_iv: "0c134f68477f2c65d311f57fb450b957"
server_key_version: "encryptionkey@0"
I20260925 04:45:10.517789 10111 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260925 04:45:10.520439 10125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:45:10.521714 10111 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 04:45:10.521852 10111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311449873301-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311449873301-393-0/minicluster-data/master-0/wal
uuid: "49cbe4adf79c47ec931a003dbf520e3d"
format_stamp: "Formatted at 2026-09-25 04:45:10 on dist-test-slave-kghz"
server_key: "c01b4dff4cda0c8e8b9351e8fe79e844"
server_key_iv: "0c134f68477f2c65d311f57fb450b957"
server_key_version: "encryptionkey@0"
I20260925 04:45:10.521968 10111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311449873301-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311449873301-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311449873301-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:45:10.526755 10111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:45:10.527395 10111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:45:10.527590 10111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:45:10.534638 10111 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:37607
I20260925 04:45:10.534674 10177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:37607 every 8 connection(s)
I20260925 04:45:10.535609 10111 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
I20260925 04:45:10.536726 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 10111
I20260925 04:45:10.536852 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311449873301-393-0/minicluster-data/master-0/wal/instance
I20260925 04:45:10.537165 393 external_mini_cluster.cc:1428] Setting key ea3167d566f026a4a1b97bc2d453c26e
I20260925 04:45:10.539541 10178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:45:10.543087 10181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:45:10.538674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39130 (local address 127.0.98.126:37607)
0925 04:45:10.539471 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0925 04:45:10.539498 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0925 04:45:10.539544 (+ 46us) server_negotiation.cc:407] Connection header received
0925 04:45:10.540293 (+ 749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:45:10.540312 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:45:10.540672 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:45:10.541034 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:45:10.541788 (+ 754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:45:10.541792 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:45:10.541804 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:45:10.541813 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:45:10.541862 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:45:10.541877 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:45:10.542036 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 04:45:10.542106 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 04:45:10.542404 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":148,"threads_started":1}
I20260925 04:45:10.545557 10178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49cbe4adf79c47ec931a003dbf520e3d: Bootstrap starting.
I20260925 04:45:10.547902 10178 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49cbe4adf79c47ec931a003dbf520e3d: Neither blocks nor log segments found. Creating new log.
I20260925 04:45:10.548637 10178 log.cc:824] T 00000000000000000000000000000000 P 49cbe4adf79c47ec931a003dbf520e3d: Log is configured to *not* fsync() on all Append() calls
I20260925 04:45:10.550670 10178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49cbe4adf79c47ec931a003dbf520e3d: No bootstrap required, opened a new log
I20260925 04:45:10.554615 10178 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49cbe4adf79c47ec931a003dbf520e3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49cbe4adf79c47ec931a003dbf520e3d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 37607 } }
I20260925 04:45:10.554893 10178 raft_consensus.cc:374] T 00000000000000000000000000000000 P 49cbe4adf79c47ec931a003dbf520e3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:45:10.554980 10178 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49cbe4adf79c47ec931a003dbf520e3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49cbe4adf79c47ec931a003dbf520e3d, State: Initialized, Role: FOLLOWER
I20260925 04:45:10.555446 10178 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49cbe4adf79c47ec931a003dbf520e3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49cbe4adf79c47ec931a003dbf520e3d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 37607 } }
I20260925 04:45:10.555599 10178 raft_consensus.cc:388] T 00000000000000000000000000000000 P 49cbe4adf79c47ec931a003dbf520e3d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:45:10.555696 10178 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49cbe4adf79c47ec931a003dbf520e3d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:45:10.555819 10178 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49cbe4adf79c47ec931a003dbf520e3d [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:45:10.556799 10178 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49cbe4adf79c47ec931a003dbf520e3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49cbe4adf79c47ec931a003dbf520e3d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 37607 } }
I20260925 04:45:10.557183 10178 leader_election.cc:302] T 00000000000000000000000000000000 P 49cbe4adf79c47ec931a003dbf520e3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 49cbe4adf79c47ec931a003dbf520e3d; no voters:
I20260925 04:45:10.557518 10178 leader_election.cc:288] T 00000000000000000000000000000000 P 49cbe4adf79c47ec931a003dbf520e3d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:45:10.557658 10183 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49cbe4adf79c47ec931a003dbf520e3d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:45:10.557942 10183 raft_consensus.cc:686] T 00000000000000000000000000000000 P 49cbe4adf79c47ec931a003dbf520e3d [term 1 LEADER]: Becoming Leader. State: Replica: 49cbe4adf79c47ec931a003dbf520e3d, State: Running, Role: LEADER
I20260925 04:45:10.558336 10183 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49cbe4adf79c47ec931a003dbf520e3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49cbe4adf79c47ec931a003dbf520e3d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 37607 } }
I20260925 04:45:10.558930 10178 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49cbe4adf79c47ec931a003dbf520e3d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:45:10.559999 10184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49cbe4adf79c47ec931a003dbf520e3d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "49cbe4adf79c47ec931a003dbf520e3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49cbe4adf79c47ec931a003dbf520e3d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 37607 } } }
I20260925 04:45:10.560127 10184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49cbe4adf79c47ec931a003dbf520e3d [sys.catalog]: This master's current role is: LEADER
I20260925 04:45:10.560587 10192 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:45:10.560729 10185 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49cbe4adf79c47ec931a003dbf520e3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49cbe4adf79c47ec931a003dbf520e3d. Latest consensus state: current_term: 1 leader_uuid: "49cbe4adf79c47ec931a003dbf520e3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49cbe4adf79c47ec931a003dbf520e3d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 37607 } } }
I20260925 04:45:10.560812 10185 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49cbe4adf79c47ec931a003dbf520e3d [sys.catalog]: This master's current role is: LEADER
I20260925 04:45:10.563886 10192 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:45:10.569752 10192 catalog_manager.cc:1347] Generated new cluster ID: 57be280af0c840e5ac5d24b527d7aaea
I20260925 04:45:10.569845 10192 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:45:10.584851 10192 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:45:10.585238 10192 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:45:10.592706 10192 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 49cbe4adf79c47ec931a003dbf520e3d: Generated new TSK 0
I20260925 04:45:10.593477 10192 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:45:10.600093 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311449873301-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311449873301-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311449873301-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:0
--local_ip_for_outbound_sockets=127.0.98.65
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:37607
--builtin_ntp_servers=127.0.98.84:41303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 04:45:10.719853 10202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:45:10.720144 10202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:45:10.720185 10202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:45:10.724365 10202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:45:10.724431 10202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:45:10.724490 10202 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260925 04:45:10.729511 10202 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:41303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311449873301-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311449873301-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:37607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10202
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311449873301-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:45:10.730674 10202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:45:10.732009 10202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:45:10.741990 10207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:45:10.741837 10210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:45:10.741842 10208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:45:10.741894 10202 server_base.cc:1034] running on GCE node
I20260925 04:45:10.742686 10202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:45:10.743407 10202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:45:10.744622 10202 hybrid_clock.cc:648] HybridClock initialized: now 1790311510744606 us; error 37 us; skew 500 ppm
I20260925 04:45:10.746852 10202 webserver.cc:460] Webserver started at http://127.0.98.65:35079/ using document root <none> and password file <none>
I20260925 04:45:10.747717 10202 fs_manager.cc:359] Metadata directory not provided
I20260925 04:45:10.747843 10202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:45:10.748098 10202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:45:10.750382 10202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311449873301-393-0/minicluster-data/ts-0/data/instance:
uuid: "2ba3f65926064a74b3973df99677d96e"
format_stamp: "Formatted at 2026-09-25 04:45:10 on dist-test-slave-kghz"
server_key: "b3086e06f0d7ea03d794fc271ced2235"
server_key_iv: "d0abb4250a3d2661a7bae32faff57fb3"
server_key_version: "encryptionkey@0"
I20260925 04:45:10.750959 10202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311449873301-393-0/minicluster-data/ts-0/wal/instance:
uuid: "2ba3f65926064a74b3973df99677d96e"
format_stamp: "Formatted at 2026-09-25 04:45:10 on dist-test-slave-kghz"
server_key: "b3086e06f0d7ea03d794fc271ced2235"
server_key_iv: "d0abb4250a3d2661a7bae32faff57fb3"
server_key_version: "encryptionkey@0"
I20260925 04:45:10.754837 10202 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 04:45:10.757382 10216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:45:10.758630 10202 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 04:45:10.758769 10202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311449873301-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311449873301-393-0/minicluster-data/ts-0/wal
uuid: "2ba3f65926064a74b3973df99677d96e"
format_stamp: "Formatted at 2026-09-25 04:45:10 on dist-test-slave-kghz"
server_key: "b3086e06f0d7ea03d794fc271ced2235"
server_key_iv: "d0abb4250a3d2661a7bae32faff57fb3"
server_key_version: "encryptionkey@0"
I20260925 04:45:10.758888 10202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311449873301-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311449873301-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311449873301-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:45:10.763947 10202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:45:10.764572 10202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:45:10.764765 10202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:45:10.765324 10202 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:45:10.766417 10202 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:45:10.766489 10202 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:45:10.766566 10202 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:45:10.766615 10202 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:45:10.775307 10202 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:43103
I20260925 04:45:10.775326 10329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:43103 every 8 connection(s)
I20260925 04:45:10.776242 10202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
I20260925 04:45:10.777065 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 10202
I20260925 04:45:10.777186 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311449873301-393-0/minicluster-data/ts-0/wal/instance
I20260925 04:45:10.777503 393 external_mini_cluster.cc:1428] Setting key 9922442cdafdc029fdbed60d36c7081f
I20260925 04:45:10.779692 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311449873301-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311449873301-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311449873301-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:0
--local_ip_for_outbound_sockets=127.0.98.66
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:37607
--builtin_ntp_servers=127.0.98.84:41303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260925 04:45:10.783365 10181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:45:10.778324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:52567 (local address 127.0.98.126:37607)
0925 04:45:10.778448 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 04:45:10.778453 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:45:10.779836 (+ 1383us) server_negotiation.cc:407] Connection header received
0925 04:45:10.781484 (+ 1648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:45:10.781486 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:45:10.781523 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:45:10.781587 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:45:10.782924 (+ 1337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:45:10.782927 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:45:10.782929 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:45:10.782940 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:45:10.782960 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:45:10.782968 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:45:10.783046 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 04:45:10.783219 (+ 173us) server_negotiation.cc:299] Negotiation successful
0925 04:45:10.783258 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 04:45:10.784662 10332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:45:10.778874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:37607 (local address 127.0.98.65:52567)
0925 04:45:10.779675 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0925 04:45:10.779717 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 04:45:10.781235 (+ 1518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:45:10.781795 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:45:10.781806 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:45:10.781939 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:45:10.782610 (+ 671us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:45:10.782615 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:45:10.782672 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:45:10.782763 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:45:10.783056 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:45:10.783067 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:45:10.783076 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 04:45:10.783266 (+ 190us) client_negotiation.cc:240] Negotiation successful
0925 04:45:10.783543 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":142,"threads_started":1}
I20260925 04:45:10.786059 10330 heartbeater.cc:347] Connected to a master server at 127.0.98.126:37607
I20260925 04:45:10.786377 10330 heartbeater.cc:464] Registering TS with master...
I20260925 04:45:10.786835 10330 heartbeater.cc:511] Master 127.0.98.126:37607 requested a full tablet report, sending...
I20260925 04:45:10.788434 10142 ts_manager.cc:194] Registered new tserver with Master: 2ba3f65926064a74b3973df99677d96e (127.0.98.65:43103)
W20260925 04:45:10.898540 10333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:45:10.898907 10333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:45:10.898986 10333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:45:10.903354 10333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:45:10.903445 10333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:45:10.903542 10333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260925 04:45:10.908562 10333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:41303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311449873301-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311449873301-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:37607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311449873301-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:45:10.909755 10333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:45:10.911309 10333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:45:10.921422 10338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:45:10.921631 10333 server_base.cc:1034] running on GCE node
W20260925 04:45:10.921422 10339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:45:10.921417 10341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:45:10.922192 10333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:45:10.922948 10333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:45:10.924144 10333 hybrid_clock.cc:648] HybridClock initialized: now 1790311510924125 us; error 32 us; skew 500 ppm
I20260925 04:45:10.926595 10333 webserver.cc:460] Webserver started at http://127.0.98.66:44203/ using document root <none> and password file <none>
I20260925 04:45:10.927234 10333 fs_manager.cc:359] Metadata directory not provided
I20260925 04:45:10.927331 10333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:45:10.927580 10333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:45:10.929921 10333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311449873301-393-0/minicluster-data/ts-1/data/instance:
uuid: "88135ee172df4f8db621c2bf8ab4a052"
format_stamp: "Formatted at 2026-09-25 04:45:10 on dist-test-slave-kghz"
server_key: "9e0a36966583eb4d6c4fa841431e1ff6"
server_key_iv: "17553d58dcb3a4c00e2a92ada5dea3e3"
server_key_version: "encryptionkey@0"
I20260925 04:45:10.930579 10333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311449873301-393-0/minicluster-data/ts-1/wal/instance:
uuid: "88135ee172df4f8db621c2bf8ab4a052"
format_stamp: "Formatted at 2026-09-25 04:45:10 on dist-test-slave-kghz"
server_key: "9e0a36966583eb4d6c4fa841431e1ff6"
server_key_iv: "17553d58dcb3a4c00e2a92ada5dea3e3"
server_key_version: "encryptionkey@0"
I20260925 04:45:10.934594 10333 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260925 04:45:10.937378 10347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:45:10.938967 10333 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 04:45:10.939119 10333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311449873301-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311449873301-393-0/minicluster-data/ts-1/wal
uuid: "88135ee172df4f8db621c2bf8ab4a052"
format_stamp: "Formatted at 2026-09-25 04:45:10 on dist-test-slave-kghz"
server_key: "9e0a36966583eb4d6c4fa841431e1ff6"
server_key_iv: "17553d58dcb3a4c00e2a92ada5dea3e3"
server_key_version: "encryptionkey@0"
I20260925 04:45:10.939272 10333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311449873301-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311449873301-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311449873301-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:45:10.944092 10333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:45:10.944729 10333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:45:10.944916 10333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:45:10.945480 10333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:45:10.946506 10333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:45:10.946587 10333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:45:10.946653 10333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:45:10.946702 10333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:45:10.955750 10333 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:42587
I20260925 04:45:10.955772 10460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:42587 every 8 connection(s)
I20260925 04:45:10.956825 10333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
I20260925 04:45:10.964018 10181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:45:10.958966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:57007 (local address 127.0.98.126:37607)
0925 04:45:10.959111 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 04:45:10.959114 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:45:10.960301 (+ 1187us) server_negotiation.cc:407] Connection header received
0925 04:45:10.961914 (+ 1613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:45:10.961918 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:45:10.961965 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:45:10.962054 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:45:10.963499 (+ 1445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:45:10.963503 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:45:10.963506 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:45:10.963520 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:45:10.963549 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:45:10.963557 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:45:10.963662 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 04:45:10.963823 (+ 161us) server_negotiation.cc:299] Negotiation successful
0925 04:45:10.963873 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 04:45:10.964956 10463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:45:10.959311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:37607 (local address 127.0.98.66:57007)
0925 04:45:10.960126 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0925 04:45:10.960181 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 04:45:10.961676 (+ 1495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:45:10.962354 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:45:10.962362 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:45:10.962471 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:45:10.963126 (+ 655us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:45:10.963128 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:45:10.963173 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:45:10.963248 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:45:10.963658 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:45:10.963667 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:45:10.963675 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 04:45:10.963873 (+ 198us) client_negotiation.cc:240] Negotiation successful
0925 04:45:10.964139 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":105,"threads_started":1}
I20260925 04:45:10.966349 10461 heartbeater.cc:347] Connected to a master server at 127.0.98.126:37607
I20260925 04:45:10.966603 10461 heartbeater.cc:464] Registering TS with master...
I20260925 04:45:10.967046 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 10333
I20260925 04:45:10.967067 10461 heartbeater.cc:511] Master 127.0.98.126:37607 requested a full tablet report, sending...
I20260925 04:45:10.967201 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311449873301-393-0/minicluster-data/ts-1/wal/instance
I20260925 04:45:10.967492 393 external_mini_cluster.cc:1428] Setting key b4201cbc4fa9c1674665826b693435dc
I20260925 04:45:10.968369 10142 ts_manager.cc:194] Registered new tserver with Master: 88135ee172df4f8db621c2bf8ab4a052 (127.0.98.66:42587)
I20260925 04:45:10.970096 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311449873301-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311449873301-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311449873301-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:0
--local_ip_for_outbound_sockets=127.0.98.67
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:37607
--builtin_ntp_servers=127.0.98.84:41303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 04:45:11.083029 10464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:45:11.083303 10464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:45:11.083343 10464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:45:11.087379 10464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:45:11.087430 10464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:45:11.087484 10464 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260925 04:45:11.094071 10464 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:41303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311449873301-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311449873301-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:37607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10464
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311449873301-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:45:11.095629 10464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:45:11.097471 10464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:45:11.107967 10472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:45:11.107967 10469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:45:11.107995 10470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:45:11.108264 10464 server_base.cc:1034] running on GCE node
I20260925 04:45:11.108683 10464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:45:11.109390 10464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:45:11.110589 10464 hybrid_clock.cc:648] HybridClock initialized: now 1790311511110550 us; error 53 us; skew 500 ppm
I20260925 04:45:11.112577 10464 webserver.cc:460] Webserver started at http://127.0.98.67:33233/ using document root <none> and password file <none>
I20260925 04:45:11.113200 10464 fs_manager.cc:359] Metadata directory not provided
I20260925 04:45:11.113294 10464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:45:11.113525 10464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:45:11.115693 10464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311449873301-393-0/minicluster-data/ts-2/data/instance:
uuid: "79c5bb1cf33a468f90fabf5f8fc63f0f"
format_stamp: "Formatted at 2026-09-25 04:45:11 on dist-test-slave-kghz"
server_key: "987be2ffadc036bd08ce5f2fd21b5e37"
server_key_iv: "36bcc523440b8219db695c247436aab9"
server_key_version: "encryptionkey@0"
I20260925 04:45:11.116286 10464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311449873301-393-0/minicluster-data/ts-2/wal/instance:
uuid: "79c5bb1cf33a468f90fabf5f8fc63f0f"
format_stamp: "Formatted at 2026-09-25 04:45:11 on dist-test-slave-kghz"
server_key: "987be2ffadc036bd08ce5f2fd21b5e37"
server_key_iv: "36bcc523440b8219db695c247436aab9"
server_key_version: "encryptionkey@0"
I20260925 04:45:11.120301 10464 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 04:45:11.123015 10478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:45:11.124363 10464 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:45:11.124511 10464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311449873301-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311449873301-393-0/minicluster-data/ts-2/wal
uuid: "79c5bb1cf33a468f90fabf5f8fc63f0f"
format_stamp: "Formatted at 2026-09-25 04:45:11 on dist-test-slave-kghz"
server_key: "987be2ffadc036bd08ce5f2fd21b5e37"
server_key_iv: "36bcc523440b8219db695c247436aab9"
server_key_version: "encryptionkey@0"
I20260925 04:45:11.124640 10464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311449873301-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311449873301-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311449873301-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:45:11.129622 10464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:45:11.130353 10464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:45:11.130542 10464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:45:11.131111 10464 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:45:11.132050 10464 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:45:11.132119 10464 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:45:11.132196 10464 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:45:11.132246 10464 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:45:11.140915 10464 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:44349
I20260925 04:45:11.140935 10591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:44349 every 8 connection(s)
I20260925 04:45:11.141825 10464 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
I20260925 04:45:11.146798 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 10464
I20260925 04:45:11.146914 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790311449873301-393-0/minicluster-data/ts-2/wal/instance
I20260925 04:45:11.147259 393 external_mini_cluster.cc:1428] Setting key b251c8d587ea1c9722e47505f831741d
I20260925 04:45:11.148576 10181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:45:11.143790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:58147 (local address 127.0.98.126:37607)
0925 04:45:11.143952 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 04:45:11.143957 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:45:11.145022 (+ 1065us) server_negotiation.cc:407] Connection header received
0925 04:45:11.146399 (+ 1377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:45:11.146402 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:45:11.146441 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:45:11.146522 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:45:11.147958 (+ 1436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:45:11.147962 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:45:11.147964 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:45:11.147976 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:45:11.147999 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:45:11.148009 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:45:11.148110 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 04:45:11.148385 (+ 275us) server_negotiation.cc:299] Negotiation successful
0925 04:45:11.148432 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 04:45:11.149588 10594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:45:11.144178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:37607 (local address 127.0.98.67:58147)
0925 04:45:11.144863 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0925 04:45:11.144903 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 04:45:11.146113 (+ 1210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:45:11.146765 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:45:11.146775 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:45:11.146898 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:45:11.147620 (+ 722us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:45:11.147621 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:45:11.147689 (+ 68us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:45:11.147799 (+ 110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:45:11.148190 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:45:11.148202 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:45:11.148212 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 04:45:11.148415 (+ 203us) client_negotiation.cc:240] Negotiation successful
0925 04:45:11.148671 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":110,"threads_started":1}
I20260925 04:45:11.150770 10592 heartbeater.cc:347] Connected to a master server at 127.0.98.126:37607
I20260925 04:45:11.151013 10592 heartbeater.cc:464] Registering TS with master...
I20260925 04:45:11.151420 10592 heartbeater.cc:511] Master 127.0.98.126:37607 requested a full tablet report, sending...
I20260925 04:45:11.152392 10141 ts_manager.cc:194] Registered new tserver with Master: 79c5bb1cf33a468f90fabf5f8fc63f0f (127.0.98.67:44349)
I20260925 04:45:11.161389 393 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 04:45:11.164605 10181 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:45:11.163516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39144 (local address 127.0.98.126:37607)
0925 04:45:11.163641 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 04:45:11.163644 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:45:11.163867 (+ 223us) server_negotiation.cc:407] Connection header received
0925 04:45:11.163975 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:45:11.163977 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:45:11.164012 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:45:11.164081 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:45:11.164459 (+ 378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39144: BlockingRecv error: recv got EOF from 127.0.0.1:39144 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 04:45:11.164655 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 10202
I20260925 04:45:11.170239 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 10333
I20260925 04:45:11.176036 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 10464
I20260925 04:45:11.181183 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 10111
2026-09-25T04:45:11Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (839 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:45:11 dist-test-slave-kghz krb5kdc[10602](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:45:11 dist-test-slave-kghz krb5kdc[10602](info): set up 2 sockets
Sep 25 04:45:11 dist-test-slave-kghz krb5kdc[10602](info): commencing operation
krb5kdc: starting...
W20260925 04:45:12.223380 393 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/krb5kdc/test-user.keytab.
Sep 25 04:45:12 dist-test-slave-kghz krb5kdc[10602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311512, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:45:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:45:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260925 04:45:12.349167 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:42291
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:38845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:42291
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:45:12.461968 10615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:45:12.462337 10615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:45:12.462386 10615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:45:12.466503 10615 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:45:12.466562 10615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:45:12.466583 10615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:45:12.466599 10615 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:45:12.466614 10615 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:45:12.472002 10615 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:38845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:42291
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:42291
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:45:12.473009 10615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:45:12.473995 10615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 04:45:12.482750 10615 server_base.cc:1034] running on GCE node
W20260925 04:45:12.482730 10621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:45:12.482661 10620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:45:12.482717 10623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:45:12.483422 10615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:45:12.484572 10615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:45:12.485762 10615 hybrid_clock.cc:648] HybridClock initialized: now 1790311512485721 us; error 55 us; skew 500 ppm
Sep 25 04:45:12 dist-test-slave-kghz krb5kdc[10602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311512, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:45:12.493137 10615 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260925 04:45:12.494115 10615 webserver.cc:460] Webserver started at http://127.0.98.126:41319/ using document root <none> and password file <none>
I20260925 04:45:12.494721 10615 fs_manager.cc:359] Metadata directory not provided
I20260925 04:45:12.494791 10615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:45:12.495014 10615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:45:12.497140 10615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/minicluster-data/master-0/data/instance:
uuid: "bb1b856f025b47bcbbfaf18998d2f659"
format_stamp: "Formatted at 2026-09-25 04:45:12 on dist-test-slave-kghz"
server_key: "42bcb04614d18e4a06dfb4bc994f09af"
server_key_iv: "6f13b31e2114609bcbddb78454291cfb"
server_key_version: "encryptionkey@0"
I20260925 04:45:12.497725 10615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/minicluster-data/master-0/wal/instance:
uuid: "bb1b856f025b47bcbbfaf18998d2f659"
format_stamp: "Formatted at 2026-09-25 04:45:12 on dist-test-slave-kghz"
server_key: "42bcb04614d18e4a06dfb4bc994f09af"
server_key_iv: "6f13b31e2114609bcbddb78454291cfb"
server_key_version: "encryptionkey@0"
I20260925 04:45:12.501781 10615 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 04:45:12.504519 10630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:45:12.505838 10615 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 04:45:12.505978 10615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/minicluster-data/master-0/wal
uuid: "bb1b856f025b47bcbbfaf18998d2f659"
format_stamp: "Formatted at 2026-09-25 04:45:12 on dist-test-slave-kghz"
server_key: "42bcb04614d18e4a06dfb4bc994f09af"
server_key_iv: "6f13b31e2114609bcbddb78454291cfb"
server_key_version: "encryptionkey@0"
I20260925 04:45:12.506101 10615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:45:12.524286 10615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:45:12.527348 10615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:45:12.527556 10615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:45:12.534513 10615 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:42291
I20260925 04:45:12.534827 10682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:42291 every 8 connection(s)
I20260925 04:45:12.535542 10615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
I20260925 04:45:12.535760 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 10615
I20260925 04:45:12.535874 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/minicluster-data/master-0/wal/instance
I20260925 04:45:12.536229 393 external_mini_cluster.cc:1428] Setting key 68969a6c3efba4602cf59e96b3652385
I20260925 04:45:12.538810 10683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:45:12.544438 10683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb1b856f025b47bcbbfaf18998d2f659: Bootstrap starting.
Sep 25 04:45:12 dist-test-slave-kghz krb5kdc[10602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311512, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:45:12.547005 10683 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb1b856f025b47bcbbfaf18998d2f659: Neither blocks nor log segments found. Creating new log.
I20260925 04:45:12.547751 10683 log.cc:824] T 00000000000000000000000000000000 P bb1b856f025b47bcbbfaf18998d2f659: Log is configured to *not* fsync() on all Append() calls
I20260925 04:45:12.549736 10683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb1b856f025b47bcbbfaf18998d2f659: No bootstrap required, opened a new log
I20260925 04:45:12.550678 10686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:45:12.537680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42316 (local address 127.0.98.126:42291)
0925 04:45:12.538143 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0925 04:45:12.538158 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 04:45:12.538194 (+ 36us) server_negotiation.cc:407] Connection header received
0925 04:45:12.538931 (+ 737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:45:12.538955 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:45:12.539329 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:45:12.539673 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:45:12.540519 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:12.541542 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:12.542479 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:12.542968 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:12.543145 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:12.545667 (+ 2522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:45:12.545672 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:45:12.545685 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:45:12.545714 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:45:12.547591 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:45:12.548144 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:45:12.548149 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:45:12.548154 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:45:12.548270 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:45:12.548570 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:45:12.548573 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:45:12.548575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:45:12.549006 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:45:12.549222 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0925 04:45:12.549521 (+ 299us) server_negotiation.cc:299] Negotiation successful
0925 04:45:12.549861 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":134,"threads_started":1}
I20260925 04:45:12.553507 10683 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb1b856f025b47bcbbfaf18998d2f659 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb1b856f025b47bcbbfaf18998d2f659" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42291 } }
I20260925 04:45:12.553726 10683 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb1b856f025b47bcbbfaf18998d2f659 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:45:12.553798 10683 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb1b856f025b47bcbbfaf18998d2f659 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb1b856f025b47bcbbfaf18998d2f659, State: Initialized, Role: FOLLOWER
I20260925 04:45:12.554193 10683 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb1b856f025b47bcbbfaf18998d2f659 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb1b856f025b47bcbbfaf18998d2f659" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42291 } }
I20260925 04:45:12.554382 10683 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb1b856f025b47bcbbfaf18998d2f659 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:45:12.554445 10683 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb1b856f025b47bcbbfaf18998d2f659 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:45:12.554576 10683 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb1b856f025b47bcbbfaf18998d2f659 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:45:12.555495 10683 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb1b856f025b47bcbbfaf18998d2f659 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb1b856f025b47bcbbfaf18998d2f659" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42291 } }
I20260925 04:45:12.555843 10683 leader_election.cc:302] T 00000000000000000000000000000000 P bb1b856f025b47bcbbfaf18998d2f659 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb1b856f025b47bcbbfaf18998d2f659; no voters:
I20260925 04:45:12.556145 10683 leader_election.cc:288] T 00000000000000000000000000000000 P bb1b856f025b47bcbbfaf18998d2f659 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:45:12.556339 10688 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb1b856f025b47bcbbfaf18998d2f659 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:45:12.556588 10688 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb1b856f025b47bcbbfaf18998d2f659 [term 1 LEADER]: Becoming Leader. State: Replica: bb1b856f025b47bcbbfaf18998d2f659, State: Running, Role: LEADER
I20260925 04:45:12.556890 10688 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb1b856f025b47bcbbfaf18998d2f659 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb1b856f025b47bcbbfaf18998d2f659" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42291 } }
I20260925 04:45:12.557444 10683 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb1b856f025b47bcbbfaf18998d2f659 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:45:12.558557 10690 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb1b856f025b47bcbbfaf18998d2f659 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb1b856f025b47bcbbfaf18998d2f659. Latest consensus state: current_term: 1 leader_uuid: "bb1b856f025b47bcbbfaf18998d2f659" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb1b856f025b47bcbbfaf18998d2f659" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42291 } } }
I20260925 04:45:12.558681 10690 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb1b856f025b47bcbbfaf18998d2f659 [sys.catalog]: This master's current role is: LEADER
I20260925 04:45:12.559077 10697 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:45:12.559680 10689 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb1b856f025b47bcbbfaf18998d2f659 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb1b856f025b47bcbbfaf18998d2f659" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb1b856f025b47bcbbfaf18998d2f659" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42291 } } }
I20260925 04:45:12.559793 10689 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb1b856f025b47bcbbfaf18998d2f659 [sys.catalog]: This master's current role is: LEADER
I20260925 04:45:12.562362 10697 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:45:12.568156 10697 catalog_manager.cc:1347] Generated new cluster ID: 0d560e2263e842bd9749a3cb95831ea4
I20260925 04:45:12.568226 10697 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:45:12.586166 10697 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:45:12.587671 10697 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:45:12.602017 10697 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb1b856f025b47bcbbfaf18998d2f659: Generated new TSK 0
I20260925 04:45:12.602972 10697 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/krb5kdc/kudu_127.0.98.65.keytab.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/krb5kdc/kudu_127.0.98.65.keytab.
I20260925 04:45:12.641500 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:0
--local_ip_for_outbound_sockets=127.0.98.65
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:42291
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:38845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:45:12.761416 10709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:45:12.761699 10709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:45:12.761744 10709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:45:12.765872 10709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:45:12.765944 10709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:45:12.766006 10709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260925 04:45:12.775702 10709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:38845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:42291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:45:12.777251 10709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:45:12.778604 10709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:45:12.790882 10714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:45:12.790860 10715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:45:12.790856 10717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:45:12.791337 10709 server_base.cc:1034] running on GCE node
I20260925 04:45:12.791771 10709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:45:12.792482 10709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:45:12.793680 10709 hybrid_clock.cc:648] HybridClock initialized: now 1790311512793653 us; error 50 us; skew 500 ppm
Sep 25 04:45:12 dist-test-slave-kghz krb5kdc[10602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311512, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:45:12.797712 10709 init.cc:377] Logged in from keytab as kudu/127.0.98.65@KRBTEST.COM (short username kudu)
I20260925 04:45:12.799151 10709 webserver.cc:460] Webserver started at http://127.0.98.65:40623/ using document root <none> and password file <none>
I20260925 04:45:12.799999 10709 fs_manager.cc:359] Metadata directory not provided
I20260925 04:45:12.800108 10709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:45:12.800350 10709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:45:12.803882 10709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/minicluster-data/ts-0/data/instance:
uuid: "0d6e239ce786432687cf4a0d25f4408d"
format_stamp: "Formatted at 2026-09-25 04:45:12 on dist-test-slave-kghz"
server_key: "fc967b1c7549a9516aae6b1cc44ff414"
server_key_iv: "93a6b511134d6bb6d6bae98d167f3aef"
server_key_version: "encryptionkey@0"
I20260925 04:45:12.804821 10709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/minicluster-data/ts-0/wal/instance:
uuid: "0d6e239ce786432687cf4a0d25f4408d"
format_stamp: "Formatted at 2026-09-25 04:45:12 on dist-test-slave-kghz"
server_key: "fc967b1c7549a9516aae6b1cc44ff414"
server_key_iv: "93a6b511134d6bb6d6bae98d167f3aef"
server_key_version: "encryptionkey@0"
I20260925 04:45:12.810642 10709 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260925 04:45:12.813534 10724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:45:12.815274 10709 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:45:12.815425 10709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/minicluster-data/ts-0/wal
uuid: "0d6e239ce786432687cf4a0d25f4408d"
format_stamp: "Formatted at 2026-09-25 04:45:12 on dist-test-slave-kghz"
server_key: "fc967b1c7549a9516aae6b1cc44ff414"
server_key_iv: "93a6b511134d6bb6d6bae98d167f3aef"
server_key_version: "encryptionkey@0"
I20260925 04:45:12.815564 10709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:45:12.834895 10709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:45:12.838173 10709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:45:12.838423 10709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:45:12.839048 10709 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:45:12.839927 10709 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:45:12.840001 10709 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:45:12.840072 10709 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:45:12.840122 10709 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:45:12.849085 10709 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:34859
I20260925 04:45:12.849231 10837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:34859 every 8 connection(s)
I20260925 04:45:12.850065 10709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
I20260925 04:45:12.859063 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 10709
I20260925 04:45:12.859191 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/minicluster-data/ts-0/wal/instance
I20260925 04:45:12.859499 393 external_mini_cluster.cc:1428] Setting key d6bc51365f63837b40844136ee65de3e
Sep 25 04:45:12 dist-test-slave-kghz krb5kdc[10602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311512, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:45:12.864905 10686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:45:12.852121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:49857 (local address 127.0.98.126:42291)
0925 04:45:12.852262 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 04:45:12.852273 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 04:45:12.853387 (+ 1114us) server_negotiation.cc:407] Connection header received
0925 04:45:12.854396 (+ 1009us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:45:12.854399 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:45:12.854461 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:45:12.854547 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:45:12.855777 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:12.856334 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:12.857617 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:12.857859 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:12.857967 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:12.861173 (+ 3206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:45:12.861177 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:45:12.861180 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:45:12.861212 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:45:12.862769 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:45:12.863311 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:45:12.863316 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:45:12.863318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:45:12.863414 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:45:12.863759 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:45:12.863764 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:45:12.863766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:45:12.863955 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:45:12.864063 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 04:45:12.864589 (+ 526us) server_negotiation.cc:299] Negotiation successful
0925 04:45:12.864732 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 04:45:12.865823 10840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:45:12.852453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:42291 (local address 127.0.98.65:49857)
0925 04:45:12.853214 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0925 04:45:12.853256 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 04:45:12.854083 (+ 827us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:45:12.854721 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:45:12.854728 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:45:12.855100 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:45:12.855619 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:45:12.855636 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:45:12.856468 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:45:12.856476 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:45:12.857487 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:45:12.857497 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:45:12.858043 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:45:12.858047 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:45:12.858233 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:12.858922 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:45:12.858946 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:45:12.860970 (+ 2024us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:45:12.862893 (+ 1923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:45:12.862900 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:45:12.862917 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:45:12.863200 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:45:12.863509 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:45:12.863513 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:45:12.863515 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:45:12.863662 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:45:12.864062 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:45:12.864071 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:45:12.864341 (+ 270us) client_negotiation.cc:769] Sending connection context
0925 04:45:12.864538 (+ 197us) client_negotiation.cc:240] Negotiation successful
0925 04:45:12.864881 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":132,"threads_started":1}
I20260925 04:45:12.867179 10838 heartbeater.cc:347] Connected to a master server at 127.0.98.126:42291
I20260925 04:45:12.867458 10838 heartbeater.cc:464] Registering TS with master...
I20260925 04:45:12.868090 10838 heartbeater.cc:511] Master 127.0.98.126:42291 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.66@KRBTEST.COM; defaulting to no policy
I20260925 04:45:12.869880 10647 ts_manager.cc:194] Registered new tserver with Master: 0d6e239ce786432687cf4a0d25f4408d (127.0.98.65:34859)
I20260925 04:45:12.871364 10647 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.65@KRBTEST.COM'} at 127.0.98.65:49857
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/krb5kdc/kudu_127.0.98.66.keytab.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/krb5kdc/kudu_127.0.98.66.keytab.
I20260925 04:45:12.889446 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:0
--local_ip_for_outbound_sockets=127.0.98.66
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:42291
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:38845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:45:13.001657 10843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:45:13.001940 10843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:45:13.001986 10843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:45:13.006142 10843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:45:13.006212 10843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:45:13.006333 10843 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260925 04:45:13.012037 10843 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:38845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:42291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10843
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:45:13.013180 10843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:45:13.014295 10843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:45:13.024605 10851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:45:13.024735 10843 server_base.cc:1034] running on GCE node
W20260925 04:45:13.024637 10849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:45:13.024544 10848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:45:13.025333 10843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:45:13.026060 10843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:45:13.027277 10843 hybrid_clock.cc:648] HybridClock initialized: now 1790311513027241 us; error 50 us; skew 500 ppm
Sep 25 04:45:13 dist-test-slave-kghz krb5kdc[10602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311513, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:45:13.030167 10843 init.cc:377] Logged in from keytab as kudu/127.0.98.66@KRBTEST.COM (short username kudu)
I20260925 04:45:13.031178 10843 webserver.cc:460] Webserver started at http://127.0.98.66:33893/ using document root <none> and password file <none>
I20260925 04:45:13.031791 10843 fs_manager.cc:359] Metadata directory not provided
I20260925 04:45:13.031857 10843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:45:13.032081 10843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:45:13.034337 10843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/minicluster-data/ts-1/data/instance:
uuid: "810405c664c54a97a51e2fbdefc2a543"
format_stamp: "Formatted at 2026-09-25 04:45:13 on dist-test-slave-kghz"
server_key: "ed0e10765cee0a52f7fd9cfa7b28a231"
server_key_iv: "606b96fccfc002557e23fbb2cc715ac9"
server_key_version: "encryptionkey@0"
I20260925 04:45:13.034946 10843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/minicluster-data/ts-1/wal/instance:
uuid: "810405c664c54a97a51e2fbdefc2a543"
format_stamp: "Formatted at 2026-09-25 04:45:13 on dist-test-slave-kghz"
server_key: "ed0e10765cee0a52f7fd9cfa7b28a231"
server_key_iv: "606b96fccfc002557e23fbb2cc715ac9"
server_key_version: "encryptionkey@0"
I20260925 04:45:13.038985 10843 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260925 04:45:13.041548 10858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:45:13.042853 10843 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 04:45:13.042984 10843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/minicluster-data/ts-1/wal
uuid: "810405c664c54a97a51e2fbdefc2a543"
format_stamp: "Formatted at 2026-09-25 04:45:13 on dist-test-slave-kghz"
server_key: "ed0e10765cee0a52f7fd9cfa7b28a231"
server_key_iv: "606b96fccfc002557e23fbb2cc715ac9"
server_key_version: "encryptionkey@0"
I20260925 04:45:13.043095 10843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:45:13.058805 10843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:45:13.062062 10843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:45:13.062299 10843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:45:13.062932 10843 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:45:13.063778 10843 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:45:13.063824 10843 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:45:13.063894 10843 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:45:13.063932 10843 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:45:13.073282 10843 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:43567
I20260925 04:45:13.073319 10971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:43567 every 8 connection(s)
I20260925 04:45:13.074204 10843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
I20260925 04:45:13.076206 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 10843
I20260925 04:45:13.076319 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/minicluster-data/ts-1/wal/instance
I20260925 04:45:13.076612 393 external_mini_cluster.cc:1428] Setting key c7243a5c76c42078ddd7b6d05102881b
Sep 25 04:45:13 dist-test-slave-kghz krb5kdc[10602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311513, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.67@KRBTEST.COM; defaulting to no policy
I20260925 04:45:13.088951 10686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:45:13.076137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:57753 (local address 127.0.98.126:42291)
0925 04:45:13.076319 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 04:45:13.076326 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:45:13.077487 (+ 1161us) server_negotiation.cc:407] Connection header received
0925 04:45:13.078457 (+ 970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:45:13.078461 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:45:13.078522 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:45:13.078616 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:45:13.079960 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:13.080484 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:13.081814 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:13.082297 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:13.082420 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:13.084937 (+ 2517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:45:13.084940 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:45:13.084944 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:45:13.084973 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:45:13.086690 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:45:13.087174 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:45:13.087176 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:45:13.087178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:45:13.087297 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:45:13.087756 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:45:13.087759 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:45:13.087761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:45:13.087956 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:45:13.088046 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 04:45:13.088711 (+ 665us) server_negotiation.cc:299] Negotiation successful
0925 04:45:13.088824 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.67@KRBTEST.COM" created.
I20260925 04:45:13.089990 10974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:45:13.076541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:42291 (local address 127.0.98.66:57753)
0925 04:45:13.077326 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0925 04:45:13.077380 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 04:45:13.078192 (+ 812us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:45:13.078777 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:45:13.078784 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:45:13.079188 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:45:13.079794 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:45:13.079816 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:45:13.080630 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:45:13.080634 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:45:13.081593 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:45:13.081602 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:45:13.082473 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:45:13.082476 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:45:13.082627 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:13.083287 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:45:13.083304 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:45:13.084764 (+ 1460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:45:13.086801 (+ 2037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:45:13.086807 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:45:13.086821 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:45:13.087080 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:45:13.087415 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:45:13.087419 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:45:13.087421 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:45:13.087599 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:45:13.088105 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:45:13.088111 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:45:13.088451 (+ 340us) client_negotiation.cc:769] Sending connection context
0925 04:45:13.088650 (+ 199us) client_negotiation.cc:240] Negotiation successful
0925 04:45:13.088934 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":151,"threads_started":1}
I20260925 04:45:13.091288 10972 heartbeater.cc:347] Connected to a master server at 127.0.98.126:42291
I20260925 04:45:13.091576 10972 heartbeater.cc:464] Registering TS with master...
I20260925 04:45:13.092203 10972 heartbeater.cc:511] Master 127.0.98.126:42291 requested a full tablet report, sending...
I20260925 04:45:13.093292 10647 ts_manager.cc:194] Registered new tserver with Master: 810405c664c54a97a51e2fbdefc2a543 (127.0.98.66:43567)
I20260925 04:45:13.093912 10647 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.66@KRBTEST.COM'} at 127.0.98.66:57753
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/krb5kdc/kudu_127.0.98.67.keytab.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/krb5kdc/kudu_127.0.98.67.keytab.
I20260925 04:45:13.105495 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:0
--local_ip_for_outbound_sockets=127.0.98.67
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:42291
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:38845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:45:13.216646 10977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:45:13.216948 10977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:45:13.217042 10977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:45:13.221396 10977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:45:13.221481 10977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:45:13.221572 10977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260925 04:45:13.226923 10977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:38845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:42291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:45:13.227994 10977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:45:13.229017 10977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:45:13.238588 10985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:45:13.238734 10977 server_base.cc:1034] running on GCE node
W20260925 04:45:13.238595 10983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:45:13.238919 10982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:45:13.239382 10977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:45:13.240051 10977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:45:13.241257 10977 hybrid_clock.cc:648] HybridClock initialized: now 1790311513241228 us; error 51 us; skew 500 ppm
Sep 25 04:45:13 dist-test-slave-kghz krb5kdc[10602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311513, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:45:13.244059 10977 init.cc:377] Logged in from keytab as kudu/127.0.98.67@KRBTEST.COM (short username kudu)
I20260925 04:45:13.245002 10977 webserver.cc:460] Webserver started at http://127.0.98.67:38499/ using document root <none> and password file <none>
I20260925 04:45:13.245589 10977 fs_manager.cc:359] Metadata directory not provided
I20260925 04:45:13.245682 10977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:45:13.245914 10977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:45:13.248107 10977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/minicluster-data/ts-2/data/instance:
uuid: "e153c023e1b04484b6494f24e0046d09"
format_stamp: "Formatted at 2026-09-25 04:45:13 on dist-test-slave-kghz"
server_key: "fbac0a5355dd66be423a77d859212f23"
server_key_iv: "b22dda41d07c5a803bc0060ab96b2b92"
server_key_version: "encryptionkey@0"
I20260925 04:45:13.248682 10977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/minicluster-data/ts-2/wal/instance:
uuid: "e153c023e1b04484b6494f24e0046d09"
format_stamp: "Formatted at 2026-09-25 04:45:13 on dist-test-slave-kghz"
server_key: "fbac0a5355dd66be423a77d859212f23"
server_key_iv: "b22dda41d07c5a803bc0060ab96b2b92"
server_key_version: "encryptionkey@0"
I20260925 04:45:13.252655 10977 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 04:45:13.255378 10992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:45:13.256790 10977 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:45:13.256958 10977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/minicluster-data/ts-2/wal
uuid: "e153c023e1b04484b6494f24e0046d09"
format_stamp: "Formatted at 2026-09-25 04:45:13 on dist-test-slave-kghz"
server_key: "fbac0a5355dd66be423a77d859212f23"
server_key_iv: "b22dda41d07c5a803bc0060ab96b2b92"
server_key_version: "encryptionkey@0"
I20260925 04:45:13.257130 10977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:45:13.270336 10977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:45:13.273722 10977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:45:13.273938 10977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:45:13.274633 10977 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:45:13.275498 10977 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:45:13.275573 10977 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:45:13.275647 10977 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:45:13.275697 10977 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:45:13.285130 10977 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:38863
I20260925 04:45:13.285146 11105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:38863 every 8 connection(s)
I20260925 04:45:13.286046 10977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
I20260925 04:45:13.291905 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 10977
I20260925 04:45:13.292045 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/minicluster-data/ts-2/wal/instance
I20260925 04:45:13.292387 393 external_mini_cluster.cc:1428] Setting key d18620797ff74c9468105df2730b0509
Sep 25 04:45:13 dist-test-slave-kghz krb5kdc[10602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311513, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:45:13.301163 10686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:45:13.287958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:48665 (local address 127.0.98.126:42291)
0925 04:45:13.288113 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 04:45:13.288118 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:45:13.289339 (+ 1221us) server_negotiation.cc:407] Connection header received
0925 04:45:13.290215 (+ 876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:45:13.290218 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:45:13.290328 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:45:13.290417 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:45:13.291692 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:13.292485 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:13.294455 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:13.294725 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:13.294828 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:13.297545 (+ 2717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:45:13.297548 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:45:13.297551 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:45:13.297576 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:45:13.299230 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:45:13.299773 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:45:13.299776 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:45:13.299777 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:45:13.299854 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:45:13.300175 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:45:13.300177 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:45:13.300178 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:45:13.300379 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:45:13.300449 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 04:45:13.300944 (+ 495us) server_negotiation.cc:299] Negotiation successful
0925 04:45:13.301040 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 04:45:13.302152 11108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:45:13.288440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:42291 (local address 127.0.98.67:48665)
0925 04:45:13.289167 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0925 04:45:13.289212 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 04:45:13.289992 (+ 780us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:45:13.290612 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:45:13.290622 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:45:13.291009 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:45:13.291498 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:45:13.291512 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:45:13.292635 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:45:13.292638 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:45:13.294167 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:45:13.294178 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:45:13.294902 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:45:13.294906 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:45:13.295114 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:13.295784 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:45:13.295801 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:45:13.297371 (+ 1570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:45:13.299378 (+ 2007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:45:13.299384 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:45:13.299398 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:45:13.299681 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:45:13.299939 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:45:13.299942 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:45:13.299944 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:45:13.300099 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:45:13.300473 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:45:13.300478 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:45:13.300737 (+ 259us) client_negotiation.cc:769] Sending connection context
0925 04:45:13.300901 (+ 164us) client_negotiation.cc:240] Negotiation successful
0925 04:45:13.301198 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":124,"threads_started":1}
I20260925 04:45:13.303464 11106 heartbeater.cc:347] Connected to a master server at 127.0.98.126:42291
I20260925 04:45:13.303727 11106 heartbeater.cc:464] Registering TS with master...
I20260925 04:45:13.304263 11106 heartbeater.cc:511] Master 127.0.98.126:42291 requested a full tablet report, sending...
I20260925 04:45:13.305421 10647 ts_manager.cc:194] Registered new tserver with Master: e153c023e1b04484b6494f24e0046d09 (127.0.98.67:38863)
I20260925 04:45:13.306192 10647 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.67@KRBTEST.COM'} at 127.0.98.67:48665
I20260925 04:45:13.306671 393 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 04:45:13.316879 10686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:45:13.308703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42330 (local address 127.0.98.126:42291)
0925 04:45:13.308829 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 04:45:13.308833 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:45:13.309028 (+ 195us) server_negotiation.cc:407] Connection header received
0925 04:45:13.309124 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:45:13.309126 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:45:13.309190 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:45:13.309286 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:45:13.310371 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:13.311068 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:13.311956 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:13.312200 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:13.312286 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:13.313764 (+ 1478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:45:13.313768 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:45:13.313770 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:45:13.313793 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:45:13.315356 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:45:13.315729 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:45:13.315732 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:45:13.315733 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:45:13.315803 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:45:13.316049 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:45:13.316052 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:45:13.316054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:45:13.316293 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:45:13.316425 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 04:45:13.316661 (+ 236us) server_negotiation.cc:299] Negotiation successful
0925 04:45:13.316759 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 04:45:13.320261 10645 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42330:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:45:13.323268 10645 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:45:13.338389 11119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:45:13.332427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46690 (local address 127.0.98.67:38863)
0925 04:45:13.333427 (+ 1000us) server_negotiation.cc:206] Beginning negotiation
0925 04:45:13.333440 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 04:45:13.333465 (+ 25us) server_negotiation.cc:407] Connection header received
0925 04:45:13.333562 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:45:13.333570 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:45:13.333793 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:45:13.334348 (+ 555us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:45:13.334526 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:13.335718 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:13.337068 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:13.337649 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:13.337775 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:13.337935 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 04:45:13.338102 (+ 167us) server_negotiation.cc:299] Negotiation successful
0925 04:45:13.338199 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":892,"thread_start_us":117,"threads_started":1}
I20260925 04:45:13.338382 11120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:45:13.332322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:38863 (local address 127.0.0.1:46690)
0925 04:45:13.332952 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0925 04:45:13.333004 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 04:45:13.333138 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:45:13.333962 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:45:13.333970 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:45:13.333994 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:45:13.334166 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:45:13.334174 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:45:13.335877 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:45:13.335880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:45:13.336914 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:45:13.336923 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:45:13.337778 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:45:13.337782 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:45:13.337885 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:13.337920 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 04:45:13.338074 (+ 154us) client_negotiation.cc:240] Negotiation successful
0925 04:45:13.338203 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"mutex_wait_us":13,"thread_start_us":111,"threads_started":1}
I20260925 04:45:13.339743 11121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:45:13.333133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45842 (local address 127.0.98.65:34859)
0925 04:45:13.333540 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0925 04:45:13.333559 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 04:45:13.333924 (+ 365us) server_negotiation.cc:407] Connection header received
0925 04:45:13.334057 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:45:13.334062 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:45:13.334224 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:45:13.334404 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:45:13.334666 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:13.335401 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:13.336943 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:13.337786 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:13.337889 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:13.338028 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 04:45:13.339466 (+ 1438us) server_negotiation.cc:299] Negotiation successful
0925 04:45:13.339592 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":84,"threads_started":1}
I20260925 04:45:13.340026 11117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:45:13.332334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:43567 (local address 127.0.0.1:35052)
0925 04:45:13.333514 (+ 1180us) negotiation.cc:107] Waiting for socket to connect
0925 04:45:13.333529 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:45:13.333624 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:45:13.334196 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:45:13.334202 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:45:13.334230 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:45:13.334354 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:45:13.334360 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:45:13.335877 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:45:13.335880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:45:13.338064 (+ 2184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:45:13.338073 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:45:13.339615 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:45:13.339619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:45:13.339707 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:13.339729 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 04:45:13.339806 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 04:45:13.339882 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1124,"mutex_wait_us":28,"thread_start_us":96,"threads_started":1}
I20260925 04:45:13.340157 11118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:45:13.332592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:34859 (local address 127.0.0.1:45842)
0925 04:45:13.333803 (+ 1211us) negotiation.cc:107] Waiting for socket to connect
0925 04:45:13.333817 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:45:13.333921 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:45:13.334411 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:45:13.334414 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:45:13.334431 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:45:13.334547 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:45:13.334553 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:45:13.335542 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:45:13.335545 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:45:13.336826 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:45:13.336835 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:45:13.338084 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:45:13.338088 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:45:13.338195 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:13.338217 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 04:45:13.338341 (+ 124us) client_negotiation.cc:240] Negotiation successful
0925 04:45:13.339892 (+ 1551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1151,"mutex_wait_us":3,"thread_start_us":101,"threads_started":1}
I20260925 04:45:13.340467 11122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:45:13.333133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35052 (local address 127.0.98.66:43567)
0925 04:45:13.333751 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0925 04:45:13.333767 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 04:45:13.333784 (+ 17us) server_negotiation.cc:407] Connection header received
0925 04:45:13.333842 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:45:13.333845 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:45:13.334043 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:45:13.334197 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:45:13.334517 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:13.335741 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:13.338198 (+ 2457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:13.339288 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:13.339399 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:13.339552 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 04:45:13.340205 (+ 653us) server_negotiation.cc:299] Negotiation successful
0925 04:45:13.340328 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":131,"threads_started":1}
I20260925 04:45:13.341976 11040 tablet_service.cc:1467] Processing CreateTablet for tablet 9f3eebcf286f4e0db14edad90323840b (DEFAULT_TABLE table=test-table [id=de8a42f8b131440186d23df7bb6f8ba4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:45:13.343182 11040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f3eebcf286f4e0db14edad90323840b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:45:13.342865 10906 tablet_service.cc:1467] Processing CreateTablet for tablet 9f3eebcf286f4e0db14edad90323840b (DEFAULT_TABLE table=test-table [id=de8a42f8b131440186d23df7bb6f8ba4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:45:13.342882 10772 tablet_service.cc:1467] Processing CreateTablet for tablet 9f3eebcf286f4e0db14edad90323840b (DEFAULT_TABLE table=test-table [id=de8a42f8b131440186d23df7bb6f8ba4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:45:13.343792 10906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f3eebcf286f4e0db14edad90323840b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:45:13.343865 10772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f3eebcf286f4e0db14edad90323840b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:45:13.349992 11123 tablet_bootstrap.cc:492] T 9f3eebcf286f4e0db14edad90323840b P 810405c664c54a97a51e2fbdefc2a543: Bootstrap starting.
I20260925 04:45:13.351094 11124 tablet_bootstrap.cc:492] T 9f3eebcf286f4e0db14edad90323840b P 0d6e239ce786432687cf4a0d25f4408d: Bootstrap starting.
I20260925 04:45:13.351524 11125 tablet_bootstrap.cc:492] T 9f3eebcf286f4e0db14edad90323840b P e153c023e1b04484b6494f24e0046d09: Bootstrap starting.
I20260925 04:45:13.353361 11125 tablet_bootstrap.cc:654] T 9f3eebcf286f4e0db14edad90323840b P e153c023e1b04484b6494f24e0046d09: Neither blocks nor log segments found. Creating new log.
I20260925 04:45:13.353454 11124 tablet_bootstrap.cc:654] T 9f3eebcf286f4e0db14edad90323840b P 0d6e239ce786432687cf4a0d25f4408d: Neither blocks nor log segments found. Creating new log.
I20260925 04:45:13.353957 11123 tablet_bootstrap.cc:654] T 9f3eebcf286f4e0db14edad90323840b P 810405c664c54a97a51e2fbdefc2a543: Neither blocks nor log segments found. Creating new log.
I20260925 04:45:13.354840 11123 log.cc:824] T 9f3eebcf286f4e0db14edad90323840b P 810405c664c54a97a51e2fbdefc2a543: Log is configured to *not* fsync() on all Append() calls
I20260925 04:45:13.354840 11124 log.cc:824] T 9f3eebcf286f4e0db14edad90323840b P 0d6e239ce786432687cf4a0d25f4408d: Log is configured to *not* fsync() on all Append() calls
I20260925 04:45:13.354888 11125 log.cc:824] T 9f3eebcf286f4e0db14edad90323840b P e153c023e1b04484b6494f24e0046d09: Log is configured to *not* fsync() on all Append() calls
I20260925 04:45:13.356972 11124 tablet_bootstrap.cc:492] T 9f3eebcf286f4e0db14edad90323840b P 0d6e239ce786432687cf4a0d25f4408d: No bootstrap required, opened a new log
I20260925 04:45:13.357178 11125 tablet_bootstrap.cc:492] T 9f3eebcf286f4e0db14edad90323840b P e153c023e1b04484b6494f24e0046d09: No bootstrap required, opened a new log
I20260925 04:45:13.357239 11124 ts_tablet_manager.cc:1397] T 9f3eebcf286f4e0db14edad90323840b P 0d6e239ce786432687cf4a0d25f4408d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 04:45:13.357333 11123 tablet_bootstrap.cc:492] T 9f3eebcf286f4e0db14edad90323840b P 810405c664c54a97a51e2fbdefc2a543: No bootstrap required, opened a new log
I20260925 04:45:13.357386 11125 ts_tablet_manager.cc:1397] T 9f3eebcf286f4e0db14edad90323840b P e153c023e1b04484b6494f24e0046d09: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 04:45:13.357512 11123 ts_tablet_manager.cc:1397] T 9f3eebcf286f4e0db14edad90323840b P 810405c664c54a97a51e2fbdefc2a543: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.002s
I20260925 04:45:13.360749 11125 raft_consensus.cc:348] T 9f3eebcf286f4e0db14edad90323840b P e153c023e1b04484b6494f24e0046d09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e153c023e1b04484b6494f24e0046d09" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38863 } } peers { permanent_uuid: "810405c664c54a97a51e2fbdefc2a543" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43567 } } peers { permanent_uuid: "0d6e239ce786432687cf4a0d25f4408d" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34859 } }
I20260925 04:45:13.361033 11125 raft_consensus.cc:374] T 9f3eebcf286f4e0db14edad90323840b P e153c023e1b04484b6494f24e0046d09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:45:13.361088 11125 raft_consensus.cc:729] T 9f3eebcf286f4e0db14edad90323840b P e153c023e1b04484b6494f24e0046d09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e153c023e1b04484b6494f24e0046d09, State: Initialized, Role: FOLLOWER
I20260925 04:45:13.361377 11123 raft_consensus.cc:348] T 9f3eebcf286f4e0db14edad90323840b P 810405c664c54a97a51e2fbdefc2a543 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e153c023e1b04484b6494f24e0046d09" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38863 } } peers { permanent_uuid: "810405c664c54a97a51e2fbdefc2a543" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43567 } } peers { permanent_uuid: "0d6e239ce786432687cf4a0d25f4408d" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34859 } }
I20260925 04:45:13.361377 11124 raft_consensus.cc:348] T 9f3eebcf286f4e0db14edad90323840b P 0d6e239ce786432687cf4a0d25f4408d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e153c023e1b04484b6494f24e0046d09" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38863 } } peers { permanent_uuid: "810405c664c54a97a51e2fbdefc2a543" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43567 } } peers { permanent_uuid: "0d6e239ce786432687cf4a0d25f4408d" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34859 } }
I20260925 04:45:13.361654 11123 raft_consensus.cc:374] T 9f3eebcf286f4e0db14edad90323840b P 810405c664c54a97a51e2fbdefc2a543 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:45:13.361670 11124 raft_consensus.cc:374] T 9f3eebcf286f4e0db14edad90323840b P 0d6e239ce786432687cf4a0d25f4408d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:45:13.361616 11125 consensus_queue.cc:260] T 9f3eebcf286f4e0db14edad90323840b P e153c023e1b04484b6494f24e0046d09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e153c023e1b04484b6494f24e0046d09" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38863 } } peers { permanent_uuid: "810405c664c54a97a51e2fbdefc2a543" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43567 } } peers { permanent_uuid: "0d6e239ce786432687cf4a0d25f4408d" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34859 } }
I20260925 04:45:13.361716 11124 raft_consensus.cc:729] T 9f3eebcf286f4e0db14edad90323840b P 0d6e239ce786432687cf4a0d25f4408d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d6e239ce786432687cf4a0d25f4408d, State: Initialized, Role: FOLLOWER
I20260925 04:45:13.361714 11123 raft_consensus.cc:729] T 9f3eebcf286f4e0db14edad90323840b P 810405c664c54a97a51e2fbdefc2a543 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 810405c664c54a97a51e2fbdefc2a543, State: Initialized, Role: FOLLOWER
I20260925 04:45:13.362156 11124 consensus_queue.cc:260] T 9f3eebcf286f4e0db14edad90323840b P 0d6e239ce786432687cf4a0d25f4408d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e153c023e1b04484b6494f24e0046d09" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38863 } } peers { permanent_uuid: "810405c664c54a97a51e2fbdefc2a543" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43567 } } peers { permanent_uuid: "0d6e239ce786432687cf4a0d25f4408d" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34859 } }
I20260925 04:45:13.362156 11123 consensus_queue.cc:260] T 9f3eebcf286f4e0db14edad90323840b P 810405c664c54a97a51e2fbdefc2a543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e153c023e1b04484b6494f24e0046d09" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38863 } } peers { permanent_uuid: "810405c664c54a97a51e2fbdefc2a543" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43567 } } peers { permanent_uuid: "0d6e239ce786432687cf4a0d25f4408d" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34859 } }
I20260925 04:45:13.362875 11125 ts_tablet_manager.cc:1428] T 9f3eebcf286f4e0db14edad90323840b P e153c023e1b04484b6494f24e0046d09: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260925 04:45:13.363126 11123 ts_tablet_manager.cc:1428] T 9f3eebcf286f4e0db14edad90323840b P 810405c664c54a97a51e2fbdefc2a543: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 04:45:13.363140 11124 ts_tablet_manager.cc:1428] T 9f3eebcf286f4e0db14edad90323840b P 0d6e239ce786432687cf4a0d25f4408d: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 04:45:13.363276 11106 heartbeater.cc:503] Master 127.0.98.126:42291 was elected leader, sending a full tablet report...
I20260925 04:45:13.363467 10972 heartbeater.cc:503] Master 127.0.98.126:42291 was elected leader, sending a full tablet report...
I20260925 04:45:13.363472 10838 heartbeater.cc:503] Master 127.0.98.126:42291 was elected leader, sending a full tablet report...
I20260925 04:45:13.535629 11129 raft_consensus.cc:482] T 9f3eebcf286f4e0db14edad90323840b P e153c023e1b04484b6494f24e0046d09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:45:13.535858 11129 raft_consensus.cc:504] T 9f3eebcf286f4e0db14edad90323840b P e153c023e1b04484b6494f24e0046d09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e153c023e1b04484b6494f24e0046d09" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38863 } } peers { permanent_uuid: "810405c664c54a97a51e2fbdefc2a543" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43567 } } peers { permanent_uuid: "0d6e239ce786432687cf4a0d25f4408d" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34859 } }
W20260925 04:45:13.537012 11107 tablet.cc:2369] T 9f3eebcf286f4e0db14edad90323840b P e153c023e1b04484b6494f24e0046d09: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:45:13.537555 11129 leader_election.cc:288] T 9f3eebcf286f4e0db14edad90323840b P e153c023e1b04484b6494f24e0046d09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 810405c664c54a97a51e2fbdefc2a543 (127.0.98.66:43567), 0d6e239ce786432687cf4a0d25f4408d (127.0.98.65:34859)
I20260925 04:45:13.541411 11108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:45:13.537313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:34859 (local address 127.0.98.67:44853)
0925 04:45:13.537506 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 04:45:13.537523 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 04:45:13.537624 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:45:13.538025 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:45:13.538028 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:45:13.538055 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:45:13.538185 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:45:13.538193 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:45:13.539167 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:45:13.539171 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:45:13.540094 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:45:13.540103 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:45:13.540988 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:45:13.540992 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:45:13.541108 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:13.541133 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 04:45:13.541205 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 04:45:13.541277 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106}
I20260925 04:45:13.541437 11121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:45:13.537625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:44853 (local address 127.0.98.65:34859)
0925 04:45:13.537765 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 04:45:13.537770 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:45:13.537783 (+ 13us) server_negotiation.cc:407] Connection header received
0925 04:45:13.537841 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:45:13.537844 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:45:13.537893 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:45:13.537980 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:45:13.538349 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:13.539057 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:13.540242 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:13.540853 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:13.540958 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:13.541021 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 04:45:13.541239 (+ 218us) server_negotiation.cc:299] Negotiation successful
0925 04:45:13.541314 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 04:45:13.542106 11122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:45:13.537668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:51587 (local address 127.0.98.66:43567)
0925 04:45:13.537805 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 04:45:13.537811 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:45:13.538360 (+ 549us) server_negotiation.cc:407] Connection header received
0925 04:45:13.538495 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:45:13.538498 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:45:13.538547 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:45:13.538626 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:45:13.538915 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:13.539661 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:13.540794 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:13.541373 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:13.541473 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:13.541548 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 04:45:13.541874 (+ 326us) server_negotiation.cc:299] Negotiation successful
0925 04:45:13.541995 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 04:45:13.542151 10792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f3eebcf286f4e0db14edad90323840b" candidate_uuid: "e153c023e1b04484b6494f24e0046d09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d6e239ce786432687cf4a0d25f4408d" is_pre_election: true
I20260925 04:45:13.542487 11132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:45:13.537313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:43567 (local address 127.0.98.67:51587)
0925 04:45:13.538229 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0925 04:45:13.538286 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 04:45:13.538386 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:45:13.538659 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:45:13.538662 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:45:13.538682 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:45:13.538793 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:45:13.538799 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:45:13.539786 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:45:13.539789 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:45:13.540667 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:45:13.540675 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:45:13.541623 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:45:13.541630 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:45:13.541735 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:13.541757 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 04:45:13.542278 (+ 521us) client_negotiation.cc:240] Negotiation successful
0925 04:45:13.542354 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":862,"mutex_wait_us":505,"thread_start_us":138,"threads_started":1}
I20260925 04:45:13.542481 10792 raft_consensus.cc:2454] T 9f3eebcf286f4e0db14edad90323840b P 0d6e239ce786432687cf4a0d25f4408d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e153c023e1b04484b6494f24e0046d09 in term 0.
I20260925 04:45:13.542959 10994 leader_election.cc:302] T 9f3eebcf286f4e0db14edad90323840b P e153c023e1b04484b6494f24e0046d09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d6e239ce786432687cf4a0d25f4408d, e153c023e1b04484b6494f24e0046d09; no voters:
I20260925 04:45:13.543131 10926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f3eebcf286f4e0db14edad90323840b" candidate_uuid: "e153c023e1b04484b6494f24e0046d09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "810405c664c54a97a51e2fbdefc2a543" is_pre_election: true
I20260925 04:45:13.543246 11129 raft_consensus.cc:2781] T 9f3eebcf286f4e0db14edad90323840b P e153c023e1b04484b6494f24e0046d09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:45:13.543321 11129 raft_consensus.cc:482] T 9f3eebcf286f4e0db14edad90323840b P e153c023e1b04484b6494f24e0046d09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:45:13.543352 11129 raft_consensus.cc:3037] T 9f3eebcf286f4e0db14edad90323840b P e153c023e1b04484b6494f24e0046d09 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:45:13.543395 10926 raft_consensus.cc:2454] T 9f3eebcf286f4e0db14edad90323840b P 810405c664c54a97a51e2fbdefc2a543 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e153c023e1b04484b6494f24e0046d09 in term 0.
I20260925 04:45:13.544456 11129 raft_consensus.cc:504] T 9f3eebcf286f4e0db14edad90323840b P e153c023e1b04484b6494f24e0046d09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e153c023e1b04484b6494f24e0046d09" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38863 } } peers { permanent_uuid: "810405c664c54a97a51e2fbdefc2a543" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43567 } } peers { permanent_uuid: "0d6e239ce786432687cf4a0d25f4408d" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34859 } }
I20260925 04:45:13.544828 11129 leader_election.cc:288] T 9f3eebcf286f4e0db14edad90323840b P e153c023e1b04484b6494f24e0046d09 [CANDIDATE]: Term 1 election: Requested vote from peers 810405c664c54a97a51e2fbdefc2a543 (127.0.98.66:43567), 0d6e239ce786432687cf4a0d25f4408d (127.0.98.65:34859)
I20260925 04:45:13.545142 10926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f3eebcf286f4e0db14edad90323840b" candidate_uuid: "e153c023e1b04484b6494f24e0046d09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "810405c664c54a97a51e2fbdefc2a543"
I20260925 04:45:13.545182 10792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f3eebcf286f4e0db14edad90323840b" candidate_uuid: "e153c023e1b04484b6494f24e0046d09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d6e239ce786432687cf4a0d25f4408d"
I20260925 04:45:13.545285 10926 raft_consensus.cc:3037] T 9f3eebcf286f4e0db14edad90323840b P 810405c664c54a97a51e2fbdefc2a543 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:45:13.545322 10792 raft_consensus.cc:3037] T 9f3eebcf286f4e0db14edad90323840b P 0d6e239ce786432687cf4a0d25f4408d [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:45:13.546406 10926 raft_consensus.cc:2454] T 9f3eebcf286f4e0db14edad90323840b P 810405c664c54a97a51e2fbdefc2a543 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e153c023e1b04484b6494f24e0046d09 in term 1.
I20260925 04:45:13.546576 10792 raft_consensus.cc:2454] T 9f3eebcf286f4e0db14edad90323840b P 0d6e239ce786432687cf4a0d25f4408d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e153c023e1b04484b6494f24e0046d09 in term 1.
I20260925 04:45:13.546710 10993 leader_election.cc:302] T 9f3eebcf286f4e0db14edad90323840b P e153c023e1b04484b6494f24e0046d09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 810405c664c54a97a51e2fbdefc2a543, e153c023e1b04484b6494f24e0046d09; no voters:
I20260925 04:45:13.546964 11129 raft_consensus.cc:2781] T 9f3eebcf286f4e0db14edad90323840b P e153c023e1b04484b6494f24e0046d09 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:45:13.547206 11129 raft_consensus.cc:686] T 9f3eebcf286f4e0db14edad90323840b P e153c023e1b04484b6494f24e0046d09 [term 1 LEADER]: Becoming Leader. State: Replica: e153c023e1b04484b6494f24e0046d09, State: Running, Role: LEADER
I20260925 04:45:13.547533 11129 consensus_queue.cc:237] T 9f3eebcf286f4e0db14edad90323840b P e153c023e1b04484b6494f24e0046d09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e153c023e1b04484b6494f24e0046d09" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38863 } } peers { permanent_uuid: "810405c664c54a97a51e2fbdefc2a543" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43567 } } peers { permanent_uuid: "0d6e239ce786432687cf4a0d25f4408d" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34859 } }
I20260925 04:45:13.550616 10647 catalog_manager.cc:5495] T 9f3eebcf286f4e0db14edad90323840b P e153c023e1b04484b6494f24e0046d09 reported cstate change: term changed from 0 to 1, leader changed from <none> to e153c023e1b04484b6494f24e0046d09 (127.0.98.67). New cstate: current_term: 1 leader_uuid: "e153c023e1b04484b6494f24e0046d09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e153c023e1b04484b6494f24e0046d09" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "810405c664c54a97a51e2fbdefc2a543" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d6e239ce786432687cf4a0d25f4408d" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34859 } health_report { overall_health: UNKNOWN } } }
I20260925 04:45:13.574620 11119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:45:13.571048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46704 (local address 127.0.98.67:38863)
0925 04:45:13.571184 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 04:45:13.571189 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:45:13.571204 (+ 15us) server_negotiation.cc:407] Connection header received
0925 04:45:13.571338 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:45:13.571340 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:45:13.571393 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:45:13.571496 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:45:13.571753 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:13.572266 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:13.573139 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:13.573359 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:13.573466 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:13.573774 (+ 308us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:45:13.574184 (+ 410us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:45:13.574343 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 04:45:13.574405 (+ 62us) server_negotiation.cc:299] Negotiation successful
0925 04:45:13.574463 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260925 04:45:13.575364 10973 tablet.cc:2369] T 9f3eebcf286f4e0db14edad90323840b P 810405c664c54a97a51e2fbdefc2a543: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:45:13.580542 10792 raft_consensus.cc:1261] T 9f3eebcf286f4e0db14edad90323840b P 0d6e239ce786432687cf4a0d25f4408d [term 1 FOLLOWER]: Refusing update from remote peer e153c023e1b04484b6494f24e0046d09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:45:13.580542 10926 raft_consensus.cc:1261] T 9f3eebcf286f4e0db14edad90323840b P 810405c664c54a97a51e2fbdefc2a543 [term 1 FOLLOWER]: Refusing update from remote peer e153c023e1b04484b6494f24e0046d09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:45:13.581292 11129 consensus_queue.cc:1035] T 9f3eebcf286f4e0db14edad90323840b P e153c023e1b04484b6494f24e0046d09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "810405c664c54a97a51e2fbdefc2a543" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:45:13.581497 11133 consensus_queue.cc:1035] T 9f3eebcf286f4e0db14edad90323840b P e153c023e1b04484b6494f24e0046d09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d6e239ce786432687cf4a0d25f4408d" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:45:13.589438 11138 mvcc.cc:205] Tried to move back new op lower bound from 7333115959617966080 to 7333115959493459968. Current Snapshot: MvccSnapshot[applied={T|T < 7333115959617966080}]
I20260925 04:45:13.600420 10646 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:45:13.600585 10646 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:45:13.602947 10646 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bb1b856f025b47bcbbfaf18998d2f659: Sending DeleteTablet for 3 replicas of tablet 9f3eebcf286f4e0db14edad90323840b
I20260925 04:45:13.603801 11040 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f3eebcf286f4e0db14edad90323840b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:45:13 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:46690
I20260925 04:45:13.603858 10772 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f3eebcf286f4e0db14edad90323840b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:45:13 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:45842
I20260925 04:45:13.604328 10906 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f3eebcf286f4e0db14edad90323840b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:45:13 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:35052
I20260925 04:45:13.604390 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 10709
I20260925 04:45:13.604628 11146 tablet_replica.cc:330] T 9f3eebcf286f4e0db14edad90323840b P e153c023e1b04484b6494f24e0046d09: stopping tablet replica
I20260925 04:45:13.605156 11148 tablet_replica.cc:330] T 9f3eebcf286f4e0db14edad90323840b P 810405c664c54a97a51e2fbdefc2a543: stopping tablet replica
I20260925 04:45:13.605487 11148 raft_consensus.cc:2229] T 9f3eebcf286f4e0db14edad90323840b P 810405c664c54a97a51e2fbdefc2a543 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:45:13.605795 11148 raft_consensus.cc:2258] T 9f3eebcf286f4e0db14edad90323840b P 810405c664c54a97a51e2fbdefc2a543 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:45:13.606717 11146 raft_consensus.cc:2229] T 9f3eebcf286f4e0db14edad90323840b P e153c023e1b04484b6494f24e0046d09 [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:45:13.607206 11146 raft_consensus.cc:2258] T 9f3eebcf286f4e0db14edad90323840b P e153c023e1b04484b6494f24e0046d09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:45:13.607364 11148 ts_tablet_manager.cc:1905] T 9f3eebcf286f4e0db14edad90323840b P 810405c664c54a97a51e2fbdefc2a543: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:45:13.608708 11146 ts_tablet_manager.cc:1905] T 9f3eebcf286f4e0db14edad90323840b P e153c023e1b04484b6494f24e0046d09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:45:13.610224 11148 ts_tablet_manager.cc:1918] T 9f3eebcf286f4e0db14edad90323840b P 810405c664c54a97a51e2fbdefc2a543: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:45:13.610349 11148 log.cc:1196] T 9f3eebcf286f4e0db14edad90323840b P 810405c664c54a97a51e2fbdefc2a543: Deleting WAL directory at /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/minicluster-data/ts-1/wal/wals/9f3eebcf286f4e0db14edad90323840b
I20260925 04:45:13.610733 11148 ts_tablet_manager.cc:1939] T 9f3eebcf286f4e0db14edad90323840b P 810405c664c54a97a51e2fbdefc2a543: Deleting consensus metadata
I20260925 04:45:13.611954 10631 catalog_manager.cc:4841] TS 810405c664c54a97a51e2fbdefc2a543 (127.0.98.66:43567): tablet 9f3eebcf286f4e0db14edad90323840b (table test-table [id=de8a42f8b131440186d23df7bb6f8ba4]) successfully deleted
I20260925 04:45:13.612576 11146 ts_tablet_manager.cc:1918] T 9f3eebcf286f4e0db14edad90323840b P e153c023e1b04484b6494f24e0046d09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:45:13.612687 11146 log.cc:1196] T 9f3eebcf286f4e0db14edad90323840b P e153c023e1b04484b6494f24e0046d09: Deleting WAL directory at /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790311449873301-393-0/minicluster-data/ts-2/wal/wals/9f3eebcf286f4e0db14edad90323840b
I20260925 04:45:13.613034 11146 ts_tablet_manager.cc:1939] T 9f3eebcf286f4e0db14edad90323840b P e153c023e1b04484b6494f24e0046d09: Deleting consensus metadata
W20260925 04:45:13.613059 10632 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.65:34859 (error 108)
I20260925 04:45:13.614065 11118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:45:13.613769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:34859 (local address 127.0.0.1:45854)
0925 04:45:13.613878 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0925 04:45:13.613954 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.65:34859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260925 04:45:13.614171 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 10843
I20260925 04:45:13.616024 10633 catalog_manager.cc:4841] TS e153c023e1b04484b6494f24e0046d09 (127.0.98.67:38863): tablet 9f3eebcf286f4e0db14edad90323840b (table test-table [id=de8a42f8b131440186d23df7bb6f8ba4]) successfully deleted
W20260925 04:45:13.618993 10632 catalog_manager.cc:4584] TS 0d6e239ce786432687cf4a0d25f4408d (127.0.98.65:34859): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9f3eebcf286f4e0db14edad90323840b: Network error: Client connection negotiation failed: client connection to 127.0.98.65:34859: connect: Connection refused (error 111)
I20260925 04:45:13.620245 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 10977
I20260925 04:45:13.626106 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 10615
2026-09-25T04:45:13Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2446 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:45:13 dist-test-slave-kghz krb5kdc[11152](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:45:13 dist-test-slave-kghz krb5kdc[11152](info): set up 2 sockets
Sep 25 04:45:13 dist-test-slave-kghz krb5kdc[11152](info): commencing operation
krb5kdc: starting...
W20260925 04:45:14.680115 393 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/krb5kdc/test-user.keytab.
Sep 25 04:45:14 dist-test-slave-kghz krb5kdc[11152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311514, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:45:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:45:14Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/krb5kdc/rangeradmin_127.0.98.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/krb5kdc/rangeradmin_127.0.98.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/krb5kdc/rangerlookup_127.0.98.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/krb5kdc/rangerlookup_127.0.98.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/krb5kdc/HTTP_127.0.98.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/krb5kdc/HTTP_127.0.98.84@KRBTEST.COM.keytab.
I20260925 04:45:14.863374 393 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskHechoP/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/postgres -l logfile start
2026-09-25 04:45:17.574 UTC [11182] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 04:45:17.574 UTC [11182] LOG: listening on IPv4 address "127.0.98.84", port 53369
2026-09-25 04:45:17.578 UTC [11182] LOG: listening on Unix socket "/tmp/.s.PGSQL.53369"
2026-09-25 04:45:17.613 UTC [11185] LOG: database system was shut down at 2026-09-25 04:45:15 UTC
2026-09-25 04:45:17.618 UTC [11182] LOG: database system is ready to accept connections
I20260925 04:45:18.563160 393 mini_postgres.cc:96] Postgres bound to 53369
2026-09-25 04:45:18.569 UTC [11193] FATAL: database "slave" does not exist
127.0.98.84:53369 - accepting connections
I20260925 04:45:18.570084 393 mini_ranger.cc:161] Starting Ranger...
I20260925 04:45:18.585661 393 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 04:45:19.148113 393 mini_ranger.cc:87] Created ranger Postgres database
I20260925 04:45:19.148191 393 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/ranger-admin
2026-09-25 04:45:19,413 [I] DB FLAVOR :POSTGRES
2026-09-25 04:45:19,414 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 04:45:19,414 [I] Checking connection..
2026-09-25 04:45:19,414 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHechoP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHechoP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:53369/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 04:45:19,658 [I] Checking connection passed.
2026-09-25 04:45:19,659 [I] --------- Verifying version history table ---------
2026-09-25 04:45:19,659 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHechoP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHechoP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:53369/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 04:45:19,906 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 04:45:19,907 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHechoP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHechoP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:53369/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 04:45:20,174 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 04:45:20,174 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 04:45:20,174 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHechoP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHechoP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:53369/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskHechoP/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 04:45:20.404 UTC [11257] WARNING: there is no transaction in progress
2026-09-25 04:45:20,411 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 04:45:20,412 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHechoP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHechoP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:53369/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 04:45:20,656 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 04:45:20,656 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 04:45:20,657 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHechoP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHechoP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:53369/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 04:45:20,888 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHechoP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHechoP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:53369/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 04:45:21,134 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHechoP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHechoP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:53369/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-kghz.c.gcp-upstream.internal','N') ;"
2026-09-25 04:45:21,383 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHechoP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHechoP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:53369/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 04:45:21,636 [I] Table x_portal_user does not exist in database ranger
2026-09-25 04:45:21,637 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHechoP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHechoP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:53369/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 04:45:21,871 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 04:45:21,871 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHechoP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHechoP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:53369/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 04:45:22,115 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHechoP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHechoP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:53369/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 04:45:22,377 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 04:45:22,378 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHechoP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHechoP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:53369/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskHechoP/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 04:45:23.460 UTC [11365] WARNING: there is no transaction in progress
2026-09-25 04:45:23.478 UTC [11365] WARNING: there is no transaction in progress
2026-09-25 04:45:23.496 UTC [11365] WARNING: there is no transaction in progress
2026-09-25 04:45:23.514 UTC [11365] WARNING: there is no transaction in progress
2026-09-25 04:45:23.534 UTC [11365] WARNING: there is no transaction in progress
2026-09-25 04:45:23.556 UTC [11365] WARNING: there is no transaction in progress
2026-09-25 04:45:23.627 UTC [11365] WARNING: there is no transaction in progress
2026-09-25 04:45:23.636 UTC [11365] WARNING: there is no transaction in progress
2026-09-25 04:45:23.653 UTC [11365] WARNING: there is no transaction in progress
2026-09-25 04:45:23.664 UTC [11365] WARNING: there is no transaction in progress
2026-09-25 04:45:23.675 UTC [11365] WARNING: there is no transaction in progress
2026-09-25 04:45:23.691 UTC [11365] WARNING: there is no transaction in progress
2026-09-25 04:45:23.702 UTC [11365] WARNING: there is no transaction in progress
2026-09-25 04:45:23.709 UTC [11365] WARNING: there is no transaction in progress
2026-09-25 04:45:23.717 UTC [11365] WARNING: there is no transaction in progress
2026-09-25 04:45:23.724 UTC [11365] WARNING: there is no transaction in progress
2026-09-25 04:45:24,432 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 04:45:24,432 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHechoP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHechoP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:53369/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 04:45:24,688 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 04:45:24,689 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHechoP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHechoP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:53369/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 04:45:24,950 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 04:45:24,950 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHechoP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHechoP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:53369/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 04:45:25,208 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 04:45:25,208 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHechoP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHechoP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:53369/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 04:45:25,441 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHechoP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHechoP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:53369/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 04:45:25,706 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHechoP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHechoP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:53369/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-kghz.c.gcp-upstream.internal';"
2026-09-25 04:45:25,938 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 04:45:25,939 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHechoP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHechoP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:53369/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 04:45:26,188 [I] DB_PATCHES have already been applied
I20260925 04:45:26.194634 393 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/ranger-admin:/tmp/dist-test-taskHechoP/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskHechoP/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskHechoP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHechoP/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 04:45:26.194705 393 mini_ranger.cc:193] Using host: 127.0.98.84
I20260925 04:45:26.198582 393 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.84:38831
Sep 25, 2026 4:45:26 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 4:45:26 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 4:45:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 4:45:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 4:45:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 4:45:27 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 4:45:27 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 04:45:27.361704 393 mini_ranger.cc:160] Time spent starting Ranger: real 8.792s user 0.002s sys 0.007s
Sep 25, 2026 4:45:47 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 4:45:47 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 25, 2026 4:45:47 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 4:45:47 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 4:45:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 4:45:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 4:45:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 4:45:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 4:45:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 4:45:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 4:45:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 4:45:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 4:45:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 4:45:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 4:45:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 4:45:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 4:45:47 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 4:45:48 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 04:45:49.279676 393 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/krb5kdc/rangerkms_127.0.98.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/krb5kdc/rangerkms_127.0.98.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/krb5kdc/HTTP_127.0.98.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/krb5kdc/HTTP_127.0.98.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 04:45:49 dist-test-slave-kghz krb5kdc[11152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311549, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 04:45:49.369997 393 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 04:45:49.379720 393 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 04:45:50.362232 393 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 04:45:50.362358 393 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/ranger-kms
I20260925 04:45:50.362457 393 mini_ranger_kms.cc:227] Using postgres at 127.0.98.84:53369
2026-09-25 04:45:50,583 [I] DB FLAVOR :POSTGRES
2026-09-25 04:45:50,583 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 04:45:50,583 [I] Checking connection
2026-09-25 04:45:50,583 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHechoP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:53369/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 04:45:50,852 [I] connection success
2026-09-25 04:45:50,852 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 04:45:50,852 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 04:45:50,852 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHechoP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:53369/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 04:45:51,121 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 04:45:51,121 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 04:45:51,122 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 04:45:51,122 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHechoP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:53369/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 04:45:51,430 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 04:45:51.907783 393 mini_ranger_kms.cc:310] Created kms service
I20260925 04:45:52.093851 393 mini_ranger_kms.cc:326] Created kudu user
I20260925 04:45:52.137494 393 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 04:45:52 dist-test-slave-kghz krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311549, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.84@KRBTEST.COM
I20260925 04:45:52.540907 393 mini_ranger_kms.cc:384] Added ranger policy
I20260925 04:45:52.541076 393 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/ranger-kms:/tmp/dist-test-taskHechoP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHechoP/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskHechoP/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 04:45:52.541103 393 mini_ranger_kms.cc:240] Using host: 127.0.98.84
I20260925 04:45:52.544667 393 mini_ranger_kms.cc:276] Ranger KMS PID: 11545
I20260925 04:45:52.544811 393 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.84:48195
Sep 25, 2026 4:45:53 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 4:45:53 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 4:45:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 4:45:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 4:45:54 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 4:45:54 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260925 04:45:54.820688 393 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.451s user 0.008s sys 0.197s
I20260925 04:45:54.820834 393 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 04:45:54.820940 393 mini_ranger_kms.cc:401] 127.0.98.84:48195/kms/v1/keys
Sep 25 04:45:55 dist-test-slave-kghz krb5kdc[11152](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790311555, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 25 04:45:55 dist-test-slave-kghz krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311555, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.84@KRBTEST.COM for HTTP/127.0.98.84@KRBTEST.COM
Sep 25 04:45:55 dist-test-slave-kghz krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311555, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.84@KRBTEST.COM for HTTP/127.0.98.84@KRBTEST.COM
Sep 25 04:45:58 dist-test-slave-kghz krb5kdc[11152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260925 04:45:58.170163 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:39355
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:40925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:39355
--ranger_config_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.84:48195/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:45:58.284070 11596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:45:58.284368 11596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:45:58.284438 11596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:45:58.288571 11596 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:45:58.288661 11596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:45:58.288699 11596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:45:58.288760 11596 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:45:58.288794 11596 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:45:58.294318 11596 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:40925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.84:48195/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:39355
--ranger_config_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:39355
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11596
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:45:58.295410 11596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:45:58.296478 11596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:45:58.305702 11601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:45:58.305827 11602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:45:58.305907 11596 server_base.cc:1034] running on GCE node
W20260925 04:45:58.305981 11604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:45:58.306589 11596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:45:58.307773 11596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:45:58.308954 11596 hybrid_clock.cc:648] HybridClock initialized: now 1790311558308928 us; error 48 us; skew 500 ppm
Sep 25 04:45:58 dist-test-slave-kghz krb5kdc[11152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311558, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:45:58.315352 11596 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260925 04:45:58.316484 11596 webserver.cc:460] Webserver started at http://127.0.98.126:44081/ using document root <none> and password file <none>
I20260925 04:45:58.317052 11596 fs_manager.cc:359] Metadata directory not provided
I20260925 04:45:58.317107 11596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:45:58.317376 11596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 04:45:58 dist-test-slave-kghz krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311558, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for HTTP/127.0.98.84@KRBTEST.COM
I20260925 04:45:58.378564 11596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/master-0/data/instance:
uuid: "330d143c128c4151a9c07256b305061d"
format_stamp: "Formatted at 2026-09-25 04:45:58 on dist-test-slave-kghz"
server_key: "dfc03754006d84fa1576ad8ee4e27081"
server_key_iv: "c9f355095a6760660129d71ae461e585"
server_key_version: "kuduclusterkey@0"
I20260925 04:45:58.379182 11596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/master-0/wal/instance:
uuid: "330d143c128c4151a9c07256b305061d"
format_stamp: "Formatted at 2026-09-25 04:45:58 on dist-test-slave-kghz"
server_key: "dfc03754006d84fa1576ad8ee4e27081"
server_key_iv: "c9f355095a6760660129d71ae461e585"
server_key_version: "kuduclusterkey@0"
I20260925 04:45:58.383142 11596 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 04:45:58.398327 11611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:45:58.399780 11596 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 04:45:58.399888 11596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/master-0/wal
uuid: "330d143c128c4151a9c07256b305061d"
format_stamp: "Formatted at 2026-09-25 04:45:58 on dist-test-slave-kghz"
server_key: "dfc03754006d84fa1576ad8ee4e27081"
server_key_iv: "c9f355095a6760660129d71ae461e585"
server_key_version: "kuduclusterkey@0"
I20260925 04:45:58.400049 11596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:45:58.421967 11596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:45:58.425074 11596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:45:58.425282 11596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:45:58.434357 11596 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:39355
I20260925 04:45:58.434396 11663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:39355 every 8 connection(s)
I20260925 04:45:58.435480 11596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
I20260925 04:45:58.437992 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 11596
I20260925 04:45:58.438136 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/master-0/wal/instance
I20260925 04:45:58.438508 393 external_mini_cluster.cc:1428] Setting key f5ea1d7e2a47aed03f5c87a4cec85aab
I20260925 04:45:58.438895 11664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:45:58.447108 11664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 330d143c128c4151a9c07256b305061d: Bootstrap starting.
I20260925 04:45:58.449723 11664 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 330d143c128c4151a9c07256b305061d: Neither blocks nor log segments found. Creating new log.
Sep 25 04:45:58 dist-test-slave-kghz krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:45:58.450639 11664 log.cc:824] T 00000000000000000000000000000000 P 330d143c128c4151a9c07256b305061d: Log is configured to *not* fsync() on all Append() calls
I20260925 04:45:58.452716 11664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 330d143c128c4151a9c07256b305061d: No bootstrap required, opened a new log
I20260925 04:45:58.456511 11664 raft_consensus.cc:348] T 00000000000000000000000000000000 P 330d143c128c4151a9c07256b305061d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "330d143c128c4151a9c07256b305061d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39355 } }
I20260925 04:45:58.456784 11664 raft_consensus.cc:374] T 00000000000000000000000000000000 P 330d143c128c4151a9c07256b305061d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:45:58.456856 11664 raft_consensus.cc:729] T 00000000000000000000000000000000 P 330d143c128c4151a9c07256b305061d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 330d143c128c4151a9c07256b305061d, State: Initialized, Role: FOLLOWER
I20260925 04:45:58.457269 11667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:45:58.440083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43666 (local address 127.0.98.126:39355)
0925 04:45:58.441062 (+ 979us) server_negotiation.cc:206] Beginning negotiation
0925 04:45:58.441083 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 04:45:58.441133 (+ 50us) server_negotiation.cc:407] Connection header received
0925 04:45:58.442028 (+ 895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:45:58.442055 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:45:58.442553 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:45:58.442970 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:45:58.443836 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:58.445513 (+ 1677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:58.446989 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:58.447670 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:58.447831 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:58.451052 (+ 3221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:45:58.451062 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:45:58.451082 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:45:58.451118 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:45:58.453754 (+ 2636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:45:58.454317 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:45:58.454324 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:45:58.454330 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:45:58.454471 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:45:58.454811 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:45:58.454815 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:45:58.454817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:45:58.455393 (+ 576us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:45:58.455597 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0925 04:45:58.455909 (+ 312us) server_negotiation.cc:299] Negotiation successful
0925 04:45:58.456205 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":820,"thread_start_us":637,"threads_started":1}
I20260925 04:45:58.457360 11664 consensus_queue.cc:260] T 00000000000000000000000000000000 P 330d143c128c4151a9c07256b305061d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "330d143c128c4151a9c07256b305061d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39355 } }
I20260925 04:45:58.457546 11664 raft_consensus.cc:388] T 00000000000000000000000000000000 P 330d143c128c4151a9c07256b305061d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:45:58.457614 11664 raft_consensus.cc:482] T 00000000000000000000000000000000 P 330d143c128c4151a9c07256b305061d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:45:58.457762 11664 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 330d143c128c4151a9c07256b305061d [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:45:58.458916 11664 raft_consensus.cc:504] T 00000000000000000000000000000000 P 330d143c128c4151a9c07256b305061d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "330d143c128c4151a9c07256b305061d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39355 } }
I20260925 04:45:58.459270 11664 leader_election.cc:302] T 00000000000000000000000000000000 P 330d143c128c4151a9c07256b305061d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 330d143c128c4151a9c07256b305061d; no voters:
I20260925 04:45:58.459606 11664 leader_election.cc:288] T 00000000000000000000000000000000 P 330d143c128c4151a9c07256b305061d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:45:58.459911 11669 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 330d143c128c4151a9c07256b305061d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:45:58.460181 11669 raft_consensus.cc:686] T 00000000000000000000000000000000 P 330d143c128c4151a9c07256b305061d [term 1 LEADER]: Becoming Leader. State: Replica: 330d143c128c4151a9c07256b305061d, State: Running, Role: LEADER
I20260925 04:45:58.460510 11669 consensus_queue.cc:237] T 00000000000000000000000000000000 P 330d143c128c4151a9c07256b305061d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "330d143c128c4151a9c07256b305061d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39355 } }
I20260925 04:45:58.461066 11664 sys_catalog.cc:564] T 00000000000000000000000000000000 P 330d143c128c4151a9c07256b305061d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:45:58.462738 11670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 330d143c128c4151a9c07256b305061d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "330d143c128c4151a9c07256b305061d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "330d143c128c4151a9c07256b305061d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39355 } } }
I20260925 04:45:58.462944 11670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 330d143c128c4151a9c07256b305061d [sys.catalog]: This master's current role is: LEADER
I20260925 04:45:58.463212 11670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 330d143c128c4151a9c07256b305061d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 330d143c128c4151a9c07256b305061d. Latest consensus state: current_term: 1 leader_uuid: "330d143c128c4151a9c07256b305061d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "330d143c128c4151a9c07256b305061d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39355 } } }
I20260925 04:45:58.463377 11670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 330d143c128c4151a9c07256b305061d [sys.catalog]: This master's current role is: LEADER
I20260925 04:45:58.463295 11664 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 04:45:58.464291 11683 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:45:58.470839 11683 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:45:58.479835 11683 catalog_manager.cc:1347] Generated new cluster ID: 11b05680ac6340cfb6846402a4502c99
I20260925 04:45:58.479914 11683 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:45:58.514575 11683 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:45:58.516036 11683 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:45:58.525980 11683 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 330d143c128c4151a9c07256b305061d: Generated new TSK 0
I20260925 04:45:58.527241 11683 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/krb5kdc/kudu_127.0.98.65.keytab.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/krb5kdc/kudu_127.0.98.65.keytab.
I20260925 04:45:59.246726 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.84:48195/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:0
--local_ip_for_outbound_sockets=127.0.98.65
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:39355
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:40925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:45:59.393673 11709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:45:59.394052 11709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:45:59.394152 11709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:45:59.400275 11709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:45:59.400404 11709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:45:59.400506 11709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260925 04:45:59.407307 11709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:40925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.84:48195/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:39355
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:45:59.408429 11709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:45:59.409600 11709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:45:59.421670 11719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:45:59.422148 11709 server_base.cc:1034] running on GCE node
W20260925 04:45:59.422158 11717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:45:59.422204 11716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:45:59.422717 11709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:45:59.423395 11709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:45:59.426740 11709 hybrid_clock.cc:648] HybridClock initialized: now 1790311559426705 us; error 46 us; skew 500 ppm
Sep 25 04:45:59 dist-test-slave-kghz krb5kdc[11152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311559, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:45:59.430511 11709 init.cc:377] Logged in from keytab as kudu/127.0.98.65@KRBTEST.COM (short username kudu)
I20260925 04:45:59.431726 11709 webserver.cc:460] Webserver started at http://127.0.98.65:45217/ using document root <none> and password file <none>
I20260925 04:45:59.432471 11709 fs_manager.cc:359] Metadata directory not provided
I20260925 04:45:59.432561 11709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:45:59.432844 11709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 04:45:59 dist-test-slave-kghz krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311559, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for HTTP/127.0.98.84@KRBTEST.COM
I20260925 04:45:59.452280 11709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/ts-0/data/instance:
uuid: "94d2139cc2444938aeca87f7baaaf076"
format_stamp: "Formatted at 2026-09-25 04:45:59 on dist-test-slave-kghz"
server_key: "0652b231ea6865b4fca90916d71f4ebd"
server_key_iv: "7bd9c946bd152a27f73f1546fa389230"
server_key_version: "kuduclusterkey@0"
I20260925 04:45:59.453138 11709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/ts-0/wal/instance:
uuid: "94d2139cc2444938aeca87f7baaaf076"
format_stamp: "Formatted at 2026-09-25 04:45:59 on dist-test-slave-kghz"
server_key: "0652b231ea6865b4fca90916d71f4ebd"
server_key_iv: "7bd9c946bd152a27f73f1546fa389230"
server_key_version: "kuduclusterkey@0"
I20260925 04:45:59.458299 11709 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260925 04:45:59.473924 11726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:45:59.475777 11709 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 04:45:59.475916 11709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/ts-0/wal
uuid: "94d2139cc2444938aeca87f7baaaf076"
format_stamp: "Formatted at 2026-09-25 04:45:59 on dist-test-slave-kghz"
server_key: "0652b231ea6865b4fca90916d71f4ebd"
server_key_iv: "7bd9c946bd152a27f73f1546fa389230"
server_key_version: "kuduclusterkey@0"
I20260925 04:45:59.476053 11709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:45:59.505831 11709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:45:59.509824 11709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:45:59.510090 11709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:45:59.510867 11709 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:45:59.511971 11709 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:45:59.512099 11709 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:45:59.512236 11709 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:45:59.512320 11709 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:45:59.526517 11709 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:40579
I20260925 04:45:59.527681 11709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
I20260925 04:45:59.529294 11841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:40579 every 8 connection(s)
I20260925 04:45:59.535982 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 11709
I20260925 04:45:59.536113 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/ts-0/wal/instance
I20260925 04:45:59.536463 393 external_mini_cluster.cc:1428] Setting key 2c78981bc0424f9ed683233cfd356497
Sep 25 04:45:59 dist-test-slave-kghz krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311559, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.66@KRBTEST.COM; defaulting to no policy
I20260925 04:45:59.550352 11845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:45:59.531367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:38403 (local address 127.0.98.126:39355)
0925 04:45:59.531627 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0925 04:45:59.531631 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:45:59.534765 (+ 3134us) server_negotiation.cc:407] Connection header received
0925 04:45:59.536475 (+ 1710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:45:59.536481 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:45:59.536555 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:45:59.536653 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:45:59.538284 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:59.539097 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:59.540979 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:59.541378 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:59.541442 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:59.545366 (+ 3924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:45:59.545374 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:45:59.545380 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:45:59.545417 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:45:59.547558 (+ 2141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:45:59.548277 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:45:59.548283 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:45:59.548288 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:45:59.548390 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:45:59.548886 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:45:59.548890 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:45:59.548892 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:45:59.549139 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:45:59.549212 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 04:45:59.550003 (+ 791us) server_negotiation.cc:299] Negotiation successful
0925 04:45:59.550132 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":99,"threads_started":1}
I20260925 04:45:59.551328 11844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:45:59.530330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39355 (local address 127.0.98.65:38403)
0925 04:45:59.534581 (+ 4251us) negotiation.cc:107] Waiting for socket to connect
0925 04:45:59.534629 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 04:45:59.536084 (+ 1455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:45:59.536996 (+ 912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:45:59.537007 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:45:59.537392 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:45:59.538094 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:45:59.538117 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:45:59.539255 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:45:59.539261 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:45:59.540826 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:45:59.540840 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:45:59.541616 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:45:59.541619 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:45:59.541776 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:59.543501 (+ 1725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:45:59.543530 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:45:59.545123 (+ 1593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:45:59.547739 (+ 2616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:45:59.547748 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:45:59.547766 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:45:59.548144 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:45:59.548521 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:45:59.548526 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:45:59.548529 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:45:59.548752 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:45:59.549302 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:45:59.549311 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:45:59.549719 (+ 408us) client_negotiation.cc:769] Sending connection context
0925 04:45:59.549964 (+ 245us) client_negotiation.cc:240] Negotiation successful
0925 04:45:59.550319 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4028,"thread_start_us":120,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.66@KRBTEST.COM" created.
I20260925 04:45:59.552913 11842 heartbeater.cc:347] Connected to a master server at 127.0.98.126:39355
I20260925 04:45:59.553174 11842 heartbeater.cc:464] Registering TS with master...
I20260925 04:45:59.553705 11842 heartbeater.cc:511] Master 127.0.98.126:39355 requested a full tablet report, sending...
I20260925 04:45:59.556767 11628 ts_manager.cc:194] Registered new tserver with Master: 94d2139cc2444938aeca87f7baaaf076 (127.0.98.65:40579)
I20260925 04:45:59.558012 11628 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.65@KRBTEST.COM'} at 127.0.98.65:38403
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/krb5kdc/kudu_127.0.98.66.keytab.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/krb5kdc/kudu_127.0.98.66.keytab.
I20260925 04:45:59.574330 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.84:48195/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:0
--local_ip_for_outbound_sockets=127.0.98.66
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:39355
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:40925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 04:45:59 dist-test-slave-kghz krb5kdc[11152](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790311559, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 04:45:59.731818 11848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:45:59.732213 11848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:45:59.732293 11848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:45:59.738962 11848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:45:59.739060 11848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:45:59.739159 11848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260925 04:45:59.747524 11848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:40925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.84:48195/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:39355
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:45:59.749529 11848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:45:59.750983 11848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:45:59.763972 11857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:45:59.764171 11855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:45:59.764477 11848 server_base.cc:1034] running on GCE node
W20260925 04:45:59.763979 11854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:45:59.765000 11848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:45:59.765817 11848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:45:59.767006 11848 hybrid_clock.cc:648] HybridClock initialized: now 1790311559766965 us; error 59 us; skew 500 ppm
Sep 25 04:45:59 dist-test-slave-kghz krb5kdc[11152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311559, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:45:59.770792 11848 init.cc:377] Logged in from keytab as kudu/127.0.98.66@KRBTEST.COM (short username kudu)
I20260925 04:45:59.772018 11848 webserver.cc:460] Webserver started at http://127.0.98.66:36729/ using document root <none> and password file <none>
I20260925 04:45:59.772730 11848 fs_manager.cc:359] Metadata directory not provided
I20260925 04:45:59.772818 11848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:45:59.773061 11848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 04:45:59 dist-test-slave-kghz krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311559, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for HTTP/127.0.98.84@KRBTEST.COM
I20260925 04:45:59.788270 11848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/ts-1/data/instance:
uuid: "e253ae9b3a7c4fc6a959087348083508"
format_stamp: "Formatted at 2026-09-25 04:45:59 on dist-test-slave-kghz"
server_key: "ac752bf23f192a72b4a42b2de3fee5ab"
server_key_iv: "726a5ff5174de28f6fba85b8b746be88"
server_key_version: "kuduclusterkey@0"
I20260925 04:45:59.790184 11848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/ts-1/wal/instance:
uuid: "e253ae9b3a7c4fc6a959087348083508"
format_stamp: "Formatted at 2026-09-25 04:45:59 on dist-test-slave-kghz"
server_key: "ac752bf23f192a72b4a42b2de3fee5ab"
server_key_iv: "726a5ff5174de28f6fba85b8b746be88"
server_key_version: "kuduclusterkey@0"
I20260925 04:45:59.795288 11848 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260925 04:45:59.814708 11864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:45:59.816453 11848 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260925 04:45:59.816614 11848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/ts-1/wal
uuid: "e253ae9b3a7c4fc6a959087348083508"
format_stamp: "Formatted at 2026-09-25 04:45:59 on dist-test-slave-kghz"
server_key: "ac752bf23f192a72b4a42b2de3fee5ab"
server_key_iv: "726a5ff5174de28f6fba85b8b746be88"
server_key_version: "kuduclusterkey@0"
I20260925 04:45:59.816782 11848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:45:59.833580 11848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:45:59.838621 11848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:45:59.838832 11848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:45:59.839609 11848 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:45:59.840731 11848 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:45:59.840806 11848 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:45:59.840867 11848 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:45:59.840904 11848 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:45:59.852366 11848 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:43179
I20260925 04:45:59.853600 11848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
I20260925 04:45:59.862815 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 11848
I20260925 04:45:59.862963 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/ts-1/wal/instance
I20260925 04:45:59.863291 393 external_mini_cluster.cc:1428] Setting key 865f01d8153300589e8e0107c9d4cf81
I20260925 04:45:59.867273 11977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:43179 every 8 connection(s)
WARNING: no policy specified for kudu/127.0.98.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.67@KRBTEST.COM" created.
Sep 25 04:45:59 dist-test-slave-kghz krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311559, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:45:59.910144 11845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:45:59.879511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:47529 (local address 127.0.98.126:39355)
0925 04:45:59.879660 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 04:45:59.879665 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:45:59.891026 (+ 11361us) server_negotiation.cc:407] Connection header received
0925 04:45:59.892500 (+ 1474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:45:59.892505 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:45:59.892574 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:45:59.892635 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:45:59.895418 (+ 2783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:59.896302 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:59.900895 (+ 4593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:45:59.901278 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:45:59.901415 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:59.905522 (+ 4107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:45:59.905527 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:45:59.905530 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:45:59.905567 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:45:59.907429 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:45:59.908185 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:45:59.908189 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:45:59.908190 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:45:59.908298 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:45:59.908759 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:45:59.908762 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:45:59.908763 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:45:59.908991 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:45:59.909072 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 04:45:59.909830 (+ 758us) server_negotiation.cc:299] Negotiation successful
0925 04:45:59.909972 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 04:45:59.912113 11981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:45:59.875452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39355 (local address 127.0.98.66:47529)
0925 04:45:59.890773 (+ 15321us) negotiation.cc:107] Waiting for socket to connect
0925 04:45:59.890874 (+ 101us) client_negotiation.cc:174] Beginning negotiation
0925 04:45:59.892180 (+ 1306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:45:59.893115 (+ 935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:45:59.893125 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:45:59.893532 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:45:59.895181 (+ 1649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:45:59.895209 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:45:59.896648 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:45:59.896651 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:45:59.900683 (+ 4032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:45:59.900703 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:45:59.901670 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:45:59.901674 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:45:59.901914 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:45:59.902754 (+ 840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:45:59.902780 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:45:59.905267 (+ 2487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:45:59.907584 (+ 2317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:45:59.907590 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:45:59.907612 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:45:59.908060 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:45:59.908449 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:45:59.908452 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:45:59.908455 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:45:59.908648 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:45:59.909142 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:45:59.909149 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:45:59.909539 (+ 390us) client_negotiation.cc:769] Sending connection context
0925 04:45:59.910682 (+ 1143us) client_negotiation.cc:240] Negotiation successful
0925 04:45:59.911049 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14979,"thread_start_us":200,"threads_started":1}
I20260925 04:45:59.913646 11978 heartbeater.cc:347] Connected to a master server at 127.0.98.126:39355
I20260925 04:45:59.913877 11978 heartbeater.cc:464] Registering TS with master...
I20260925 04:45:59.914438 11978 heartbeater.cc:511] Master 127.0.98.126:39355 requested a full tablet report, sending...
I20260925 04:45:59.915572 11628 ts_manager.cc:194] Registered new tserver with Master: e253ae9b3a7c4fc6a959087348083508 (127.0.98.66:43179)
I20260925 04:45:59.916299 11628 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.66@KRBTEST.COM'} at 127.0.98.66:47529
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/krb5kdc/kudu_127.0.98.67.keytab.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/krb5kdc/kudu_127.0.98.67.keytab.
I20260925 04:45:59.935259 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.84:48195/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:0
--local_ip_for_outbound_sockets=127.0.98.67
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:39355
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:40925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:46:00.101168 11983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:46:00.102016 11983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:46:00.102166 11983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:46:00.108701 11983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:46:00.108857 11983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:46:00.108991 11983 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260925 04:46:00.117233 11983 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:40925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.84:48195/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:39355
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:46:00.119019 11983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:46:00.120682 11983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:46:00.135838 11989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:46:00.136324 11983 server_base.cc:1034] running on GCE node
W20260925 04:46:00.136912 11988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:46:00.137790 11991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:46:00.138432 11983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:46:00.139262 11983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:46:00.140458 11983 hybrid_clock.cc:648] HybridClock initialized: now 1790311560140390 us; error 89 us; skew 500 ppm
Sep 25 04:46:00 dist-test-slave-kghz krb5kdc[11152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311560, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:46:00.144205 11983 init.cc:377] Logged in from keytab as kudu/127.0.98.67@KRBTEST.COM (short username kudu)
I20260925 04:46:00.145298 11983 webserver.cc:460] Webserver started at http://127.0.98.67:33361/ using document root <none> and password file <none>
I20260925 04:46:00.146005 11983 fs_manager.cc:359] Metadata directory not provided
I20260925 04:46:00.146086 11983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:46:00.146335 11983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 04:46:00 dist-test-slave-kghz krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311560, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for HTTP/127.0.98.84@KRBTEST.COM
I20260925 04:46:00.161171 11983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/ts-2/data/instance:
uuid: "a8b7389ce35d43ddb1b48891c3cbe5e0"
format_stamp: "Formatted at 2026-09-25 04:46:00 on dist-test-slave-kghz"
server_key: "754f78e060a4f065a5b2942bf8262e73"
server_key_iv: "68ef86c3bb4ffde014ade0a420335d86"
server_key_version: "kuduclusterkey@0"
I20260925 04:46:00.162124 11983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/ts-2/wal/instance:
uuid: "a8b7389ce35d43ddb1b48891c3cbe5e0"
format_stamp: "Formatted at 2026-09-25 04:46:00 on dist-test-slave-kghz"
server_key: "754f78e060a4f065a5b2942bf8262e73"
server_key_iv: "68ef86c3bb4ffde014ade0a420335d86"
server_key_version: "kuduclusterkey@0"
I20260925 04:46:00.179020 11983 fs_manager.cc:691] Time spent creating directory manager: real 0.016s user 0.004s sys 0.000s
Sep 25 04:46:00 dist-test-slave-kghz krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311559, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for HTTP/127.0.98.84@KRBTEST.COM
I20260925 04:46:00.203913 11998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:46:00.205754 11983 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.002s sys 0.000s
I20260925 04:46:00.205960 11983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/ts-2/wal
uuid: "a8b7389ce35d43ddb1b48891c3cbe5e0"
format_stamp: "Formatted at 2026-09-25 04:46:00 on dist-test-slave-kghz"
server_key: "754f78e060a4f065a5b2942bf8262e73"
server_key_iv: "68ef86c3bb4ffde014ade0a420335d86"
server_key_version: "kuduclusterkey@0"
I20260925 04:46:00.206127 11983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:46:00.255818 11983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:46:00.259817 11983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:46:00.260095 11983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:46:00.260876 11983 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:46:00.262102 11983 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:46:00.262218 11983 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:46:00.262356 11983 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:46:00.262478 11983 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:46:00.273691 11983 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:39197
I20260925 04:46:00.278061 11983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
I20260925 04:46:00.279861 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 11983
I20260925 04:46:00.279985 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/ts-2/wal/instance
I20260925 04:46:00.280290 393 external_mini_cluster.cc:1428] Setting key 5f6552ca4a8eda4f8f98be01d20c0459
I20260925 04:46:00.281088 12111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:39197 every 8 connection(s)
Sep 25 04:46:00 dist-test-slave-kghz krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311560, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:46:00.317704 11845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:00.285218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:57415 (local address 127.0.98.126:39355)
0925 04:46:00.285334 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0925 04:46:00.285338 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:46:00.294667 (+ 9329us) server_negotiation.cc:407] Connection header received
0925 04:46:00.295929 (+ 1262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:46:00.295932 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:46:00.295984 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:46:00.296032 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:46:00.303312 (+ 7280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:00.304175 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:00.307386 (+ 3211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:00.307742 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:00.307815 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:00.311179 (+ 3364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:46:00.311184 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:46:00.311186 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:46:00.311215 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:46:00.313290 (+ 2075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:46:00.314268 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:46:00.314272 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:46:00.314274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:46:00.314382 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:46:00.314799 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:46:00.314802 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:46:00.314804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:46:00.315034 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:46:00.315114 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 04:46:00.317392 (+ 2278us) server_negotiation.cc:299] Negotiation successful
0925 04:46:00.317541 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 04:46:00.320282 12114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:00.284140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39355 (local address 127.0.98.67:57415)
0925 04:46:00.294488 (+ 10348us) negotiation.cc:107] Waiting for socket to connect
0925 04:46:00.294569 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0925 04:46:00.295680 (+ 1111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:46:00.296466 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:46:00.296475 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:46:00.296835 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:46:00.297548 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:00.297566 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:00.304426 (+ 6860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:00.304430 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:00.307223 (+ 2793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:00.307237 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:00.308030 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:00.308034 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:00.308245 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:00.309063 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:46:00.309136 (+ 73us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:46:00.310950 (+ 1814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:46:00.313498 (+ 2548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:46:00.313505 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:46:00.313524 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:46:00.314116 (+ 592us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:46:00.314498 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:46:00.314501 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:46:00.314503 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:46:00.314694 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:46:00.315180 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:46:00.315187 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:46:00.315567 (+ 380us) client_negotiation.cc:769] Sending connection context
0925 04:46:00.318812 (+ 3245us) client_negotiation.cc:240] Negotiation successful
0925 04:46:00.319197 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9406,"thread_start_us":165,"threads_started":1}
I20260925 04:46:00.331080 12112 heartbeater.cc:347] Connected to a master server at 127.0.98.126:39355
I20260925 04:46:00.331347 12112 heartbeater.cc:464] Registering TS with master...
I20260925 04:46:00.331962 12112 heartbeater.cc:511] Master 127.0.98.126:39355 requested a full tablet report, sending...
I20260925 04:46:00.333786 11628 ts_manager.cc:194] Registered new tserver with Master: a8b7389ce35d43ddb1b48891c3cbe5e0 (127.0.98.67:39197)
I20260925 04:46:00.334580 11628 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.67@KRBTEST.COM'} at 127.0.98.67:57415
I20260925 04:46:00.345151 393 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 04:46:00.360292 11845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:00.349691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35342 (local address 127.0.98.126:39355)
0925 04:46:00.349826 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 04:46:00.349831 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:46:00.350218 (+ 387us) server_negotiation.cc:407] Connection header received
0925 04:46:00.350462 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:46:00.350465 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:46:00.350515 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:46:00.350568 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:46:00.351579 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:00.352427 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:00.353718 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:00.354076 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:00.354150 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:00.356104 (+ 1954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:46:00.356108 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:46:00.356110 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:46:00.356139 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:46:00.357985 (+ 1846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:46:00.358609 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:46:00.358613 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:46:00.358615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:46:00.358700 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:46:00.359014 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:46:00.359018 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:46:00.359019 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:46:00.359234 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:46:00.359347 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 04:46:00.359986 (+ 639us) server_negotiation.cc:299] Negotiation successful
0925 04:46:00.360129 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 04:46:00.364146 11628 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35342:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:46:00.367396 11628 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:46:00.400455 12127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:00.389665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:40579 (local address 127.0.0.1:57570)
0925 04:46:00.391343 (+ 1678us) negotiation.cc:107] Waiting for socket to connect
0925 04:46:00.391401 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 04:46:00.391531 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:46:00.394996 (+ 3465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:46:00.395004 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:46:00.395033 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:46:00.395194 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:00.395201 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:00.396723 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:00.396730 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:00.398507 (+ 1777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:00.398519 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:00.399879 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:00.399883 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:00.400002 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:00.400043 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 04:46:00.400149 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 04:46:00.400290 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1484,"thread_start_us":137,"threads_started":1}
I20260925 04:46:00.401225 12129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:00.390735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57570 (local address 127.0.98.65:40579)
0925 04:46:00.391799 (+ 1064us) server_negotiation.cc:206] Beginning negotiation
0925 04:46:00.391807 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:46:00.391830 (+ 23us) server_negotiation.cc:407] Connection header received
0925 04:46:00.391905 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:46:00.391910 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:46:00.392117 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:46:00.392261 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:46:00.395389 (+ 3128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:00.396577 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:00.398655 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:00.399527 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:00.399649 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:00.399812 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 04:46:00.400963 (+ 1151us) server_negotiation.cc:299] Negotiation successful
0925 04:46:00.401086 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":960,"thread_start_us":66,"threads_started":1}
I20260925 04:46:00.404866 11775 tablet_service.cc:1467] Processing CreateTablet for tablet 1f204f2b3ff94d80bef83eade1d7fc66 (DEFAULT_TABLE table=test-table [id=9780a73895494394a6486673eec22bc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:46:00.406215 11775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f204f2b3ff94d80bef83eade1d7fc66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:46:00.415144 12130 tablet_bootstrap.cc:492] T 1f204f2b3ff94d80bef83eade1d7fc66 P 94d2139cc2444938aeca87f7baaaf076: Bootstrap starting.
I20260925 04:46:00.415213 12125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:00.389708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:39197 (local address 127.0.0.1:38970)
0925 04:46:00.396680 (+ 6972us) negotiation.cc:107] Waiting for socket to connect
0925 04:46:00.396698 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 04:46:00.396794 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:46:00.403931 (+ 7137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:46:00.403935 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:46:00.403952 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:46:00.404037 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:00.404043 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:00.409059 (+ 5016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:00.409064 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:00.411380 (+ 2316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:00.411392 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:00.414183 (+ 2791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:00.414187 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:00.414299 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:00.414324 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 04:46:00.414397 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 04:46:00.414476 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6877,"thread_start_us":101,"threads_started":1}
I20260925 04:46:00.417052 12124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:00.390461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:43179 (local address 127.0.0.1:42010)
0925 04:46:00.396973 (+ 6512us) negotiation.cc:107] Waiting for socket to connect
0925 04:46:00.396986 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:46:00.397965 (+ 979us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:46:00.401939 (+ 3974us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:46:00.401942 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:46:00.401961 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:46:00.402046 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:00.402051 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:00.411530 (+ 9479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:00.411533 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:00.412837 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:00.412854 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:00.416635 (+ 3781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:00.416638 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:00.416735 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:00.416759 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 04:46:00.416843 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 04:46:00.416920 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6455,"thread_start_us":87,"threads_started":1}
I20260925 04:46:00.417569 12126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:00.389919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38970 (local address 127.0.98.67:39197)
0925 04:46:00.402358 (+ 12439us) server_negotiation.cc:206] Beginning negotiation
0925 04:46:00.402371 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 04:46:00.402394 (+ 23us) server_negotiation.cc:407] Connection header received
0925 04:46:00.402469 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:46:00.402474 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:46:00.402659 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:46:00.403781 (+ 1122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:46:00.404168 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:00.406642 (+ 2474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:00.413036 (+ 6394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:00.413849 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:00.413963 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:00.414115 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 04:46:00.417294 (+ 3179us) server_negotiation.cc:299] Negotiation successful
0925 04:46:00.417423 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12301,"thread_start_us":141,"threads_started":1}
I20260925 04:46:00.418079 12128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:00.390604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42010 (local address 127.0.98.66:43179)
0925 04:46:00.401209 (+ 10605us) server_negotiation.cc:206] Beginning negotiation
0925 04:46:00.401230 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 04:46:00.401258 (+ 28us) server_negotiation.cc:407] Connection header received
0925 04:46:00.401346 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:46:00.401355 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:46:00.401611 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:46:00.401790 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:46:00.404655 (+ 2865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:00.405743 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:00.415460 (+ 9717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:00.416289 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:00.416416 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:00.416572 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 04:46:00.417832 (+ 1260us) server_negotiation.cc:299] Negotiation successful
0925 04:46:00.417941 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10490,"thread_start_us":102,"threads_started":1}
I20260925 04:46:00.418813 12130 tablet_bootstrap.cc:654] T 1f204f2b3ff94d80bef83eade1d7fc66 P 94d2139cc2444938aeca87f7baaaf076: Neither blocks nor log segments found. Creating new log.
I20260925 04:46:00.419811 12130 log.cc:824] T 1f204f2b3ff94d80bef83eade1d7fc66 P 94d2139cc2444938aeca87f7baaaf076: Log is configured to *not* fsync() on all Append() calls
I20260925 04:46:00.421427 12046 tablet_service.cc:1467] Processing CreateTablet for tablet 1f204f2b3ff94d80bef83eade1d7fc66 (DEFAULT_TABLE table=test-table [id=9780a73895494394a6486673eec22bc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:46:00.422818 12046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f204f2b3ff94d80bef83eade1d7fc66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:46:00.423297 11912 tablet_service.cc:1467] Processing CreateTablet for tablet 1f204f2b3ff94d80bef83eade1d7fc66 (DEFAULT_TABLE table=test-table [id=9780a73895494394a6486673eec22bc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:46:00.424368 11912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f204f2b3ff94d80bef83eade1d7fc66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:46:00.431945 12133 tablet_bootstrap.cc:492] T 1f204f2b3ff94d80bef83eade1d7fc66 P e253ae9b3a7c4fc6a959087348083508: Bootstrap starting.
I20260925 04:46:00.434199 12130 tablet_bootstrap.cc:492] T 1f204f2b3ff94d80bef83eade1d7fc66 P 94d2139cc2444938aeca87f7baaaf076: No bootstrap required, opened a new log
I20260925 04:46:00.434541 12130 ts_tablet_manager.cc:1397] T 1f204f2b3ff94d80bef83eade1d7fc66 P 94d2139cc2444938aeca87f7baaaf076: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.000s
I20260925 04:46:00.434597 12132 tablet_bootstrap.cc:492] T 1f204f2b3ff94d80bef83eade1d7fc66 P a8b7389ce35d43ddb1b48891c3cbe5e0: Bootstrap starting.
I20260925 04:46:00.435039 12133 tablet_bootstrap.cc:654] T 1f204f2b3ff94d80bef83eade1d7fc66 P e253ae9b3a7c4fc6a959087348083508: Neither blocks nor log segments found. Creating new log.
I20260925 04:46:00.437879 12132 tablet_bootstrap.cc:654] T 1f204f2b3ff94d80bef83eade1d7fc66 P a8b7389ce35d43ddb1b48891c3cbe5e0: Neither blocks nor log segments found. Creating new log.
I20260925 04:46:00.439360 12133 log.cc:824] T 1f204f2b3ff94d80bef83eade1d7fc66 P e253ae9b3a7c4fc6a959087348083508: Log is configured to *not* fsync() on all Append() calls
I20260925 04:46:00.439824 12130 raft_consensus.cc:348] T 1f204f2b3ff94d80bef83eade1d7fc66 P 94d2139cc2444938aeca87f7baaaf076 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8b7389ce35d43ddb1b48891c3cbe5e0" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 39197 } } peers { permanent_uuid: "94d2139cc2444938aeca87f7baaaf076" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 40579 } } peers { permanent_uuid: "e253ae9b3a7c4fc6a959087348083508" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43179 } }
I20260925 04:46:00.440305 12130 raft_consensus.cc:374] T 1f204f2b3ff94d80bef83eade1d7fc66 P 94d2139cc2444938aeca87f7baaaf076 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:46:00.440410 12130 raft_consensus.cc:729] T 1f204f2b3ff94d80bef83eade1d7fc66 P 94d2139cc2444938aeca87f7baaaf076 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94d2139cc2444938aeca87f7baaaf076, State: Initialized, Role: FOLLOWER
I20260925 04:46:00.440779 12132 log.cc:824] T 1f204f2b3ff94d80bef83eade1d7fc66 P a8b7389ce35d43ddb1b48891c3cbe5e0: Log is configured to *not* fsync() on all Append() calls
I20260925 04:46:00.441036 12130 consensus_queue.cc:260] T 1f204f2b3ff94d80bef83eade1d7fc66 P 94d2139cc2444938aeca87f7baaaf076 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8b7389ce35d43ddb1b48891c3cbe5e0" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 39197 } } peers { permanent_uuid: "94d2139cc2444938aeca87f7baaaf076" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 40579 } } peers { permanent_uuid: "e253ae9b3a7c4fc6a959087348083508" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43179 } }
I20260925 04:46:00.442332 12130 ts_tablet_manager.cc:1428] T 1f204f2b3ff94d80bef83eade1d7fc66 P 94d2139cc2444938aeca87f7baaaf076: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260925 04:46:00.442739 11842 heartbeater.cc:503] Master 127.0.98.126:39355 was elected leader, sending a full tablet report...
I20260925 04:46:00.443163 12133 tablet_bootstrap.cc:492] T 1f204f2b3ff94d80bef83eade1d7fc66 P e253ae9b3a7c4fc6a959087348083508: No bootstrap required, opened a new log
I20260925 04:46:00.443356 12133 ts_tablet_manager.cc:1397] T 1f204f2b3ff94d80bef83eade1d7fc66 P e253ae9b3a7c4fc6a959087348083508: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.004s
I20260925 04:46:00.448002 12133 raft_consensus.cc:348] T 1f204f2b3ff94d80bef83eade1d7fc66 P e253ae9b3a7c4fc6a959087348083508 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8b7389ce35d43ddb1b48891c3cbe5e0" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 39197 } } peers { permanent_uuid: "94d2139cc2444938aeca87f7baaaf076" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 40579 } } peers { permanent_uuid: "e253ae9b3a7c4fc6a959087348083508" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43179 } }
I20260925 04:46:00.448439 12133 raft_consensus.cc:374] T 1f204f2b3ff94d80bef83eade1d7fc66 P e253ae9b3a7c4fc6a959087348083508 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:46:00.448554 12133 raft_consensus.cc:729] T 1f204f2b3ff94d80bef83eade1d7fc66 P e253ae9b3a7c4fc6a959087348083508 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e253ae9b3a7c4fc6a959087348083508, State: Initialized, Role: FOLLOWER
I20260925 04:46:00.450527 12132 tablet_bootstrap.cc:492] T 1f204f2b3ff94d80bef83eade1d7fc66 P a8b7389ce35d43ddb1b48891c3cbe5e0: No bootstrap required, opened a new log
I20260925 04:46:00.450841 12132 ts_tablet_manager.cc:1397] T 1f204f2b3ff94d80bef83eade1d7fc66 P a8b7389ce35d43ddb1b48891c3cbe5e0: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.000s
I20260925 04:46:00.455109 12132 raft_consensus.cc:348] T 1f204f2b3ff94d80bef83eade1d7fc66 P a8b7389ce35d43ddb1b48891c3cbe5e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8b7389ce35d43ddb1b48891c3cbe5e0" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 39197 } } peers { permanent_uuid: "94d2139cc2444938aeca87f7baaaf076" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 40579 } } peers { permanent_uuid: "e253ae9b3a7c4fc6a959087348083508" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43179 } }
I20260925 04:46:00.455724 12133 consensus_queue.cc:260] T 1f204f2b3ff94d80bef83eade1d7fc66 P e253ae9b3a7c4fc6a959087348083508 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8b7389ce35d43ddb1b48891c3cbe5e0" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 39197 } } peers { permanent_uuid: "94d2139cc2444938aeca87f7baaaf076" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 40579 } } peers { permanent_uuid: "e253ae9b3a7c4fc6a959087348083508" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43179 } }
I20260925 04:46:00.457106 12133 ts_tablet_manager.cc:1428] T 1f204f2b3ff94d80bef83eade1d7fc66 P e253ae9b3a7c4fc6a959087348083508: Time spent starting tablet: real 0.014s user 0.005s sys 0.002s
I20260925 04:46:00.457269 12132 raft_consensus.cc:374] T 1f204f2b3ff94d80bef83eade1d7fc66 P a8b7389ce35d43ddb1b48891c3cbe5e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:46:00.457386 12132 raft_consensus.cc:729] T 1f204f2b3ff94d80bef83eade1d7fc66 P a8b7389ce35d43ddb1b48891c3cbe5e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8b7389ce35d43ddb1b48891c3cbe5e0, State: Initialized, Role: FOLLOWER
I20260925 04:46:00.458170 12132 consensus_queue.cc:260] T 1f204f2b3ff94d80bef83eade1d7fc66 P a8b7389ce35d43ddb1b48891c3cbe5e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8b7389ce35d43ddb1b48891c3cbe5e0" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 39197 } } peers { permanent_uuid: "94d2139cc2444938aeca87f7baaaf076" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 40579 } } peers { permanent_uuid: "e253ae9b3a7c4fc6a959087348083508" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43179 } }
I20260925 04:46:00.458539 11978 heartbeater.cc:503] Master 127.0.98.126:39355 was elected leader, sending a full tablet report...
I20260925 04:46:00.462153 12132 ts_tablet_manager.cc:1428] T 1f204f2b3ff94d80bef83eade1d7fc66 P a8b7389ce35d43ddb1b48891c3cbe5e0: Time spent starting tablet: real 0.011s user 0.002s sys 0.004s
I20260925 04:46:00.462594 12112 heartbeater.cc:503] Master 127.0.98.126:39355 was elected leader, sending a full tablet report...
W20260925 04:46:00.530877 11843 tablet.cc:2369] T 1f204f2b3ff94d80bef83eade1d7fc66 P 94d2139cc2444938aeca87f7baaaf076: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 04:46:00.532109 12113 tablet.cc:2369] T 1f204f2b3ff94d80bef83eade1d7fc66 P a8b7389ce35d43ddb1b48891c3cbe5e0: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 04:46:00 dist-test-slave-kghz krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311559, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for HTTP/127.0.98.84@KRBTEST.COM
W20260925 04:46:00.619633 11979 tablet.cc:2369] T 1f204f2b3ff94d80bef83eade1d7fc66 P e253ae9b3a7c4fc6a959087348083508: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:46:00.633934 12137 raft_consensus.cc:482] T 1f204f2b3ff94d80bef83eade1d7fc66 P e253ae9b3a7c4fc6a959087348083508 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:46:00.634179 12137 raft_consensus.cc:504] T 1f204f2b3ff94d80bef83eade1d7fc66 P e253ae9b3a7c4fc6a959087348083508 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8b7389ce35d43ddb1b48891c3cbe5e0" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 39197 } } peers { permanent_uuid: "94d2139cc2444938aeca87f7baaaf076" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 40579 } } peers { permanent_uuid: "e253ae9b3a7c4fc6a959087348083508" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43179 } }
I20260925 04:46:00.635684 12137 leader_election.cc:288] T 1f204f2b3ff94d80bef83eade1d7fc66 P e253ae9b3a7c4fc6a959087348083508 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a8b7389ce35d43ddb1b48891c3cbe5e0 (127.0.98.67:39197), 94d2139cc2444938aeca87f7baaaf076 (127.0.98.65:40579)
I20260925 04:46:00.641906 12129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:00.636567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:48257 (local address 127.0.98.65:40579)
0925 04:46:00.636703 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 04:46:00.636708 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:46:00.637318 (+ 610us) server_negotiation.cc:407] Connection header received
0925 04:46:00.637551 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:46:00.637555 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:46:00.637607 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:46:00.637657 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:46:00.638098 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:00.638904 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:00.640560 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:00.641142 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:00.641230 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:00.641314 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 04:46:00.641678 (+ 364us) server_negotiation.cc:299] Negotiation successful
0925 04:46:00.641765 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 04:46:00.642606 12141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:00.636073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:40579 (local address 127.0.98.66:48257)
0925 04:46:00.637208 (+ 1135us) negotiation.cc:107] Waiting for socket to connect
0925 04:46:00.637225 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 04:46:00.637406 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:46:00.637803 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:46:00.637805 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:46:00.637837 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:46:00.637966 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:00.637973 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:00.639057 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:00.639061 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:00.640411 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:00.640423 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:00.641418 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:00.641423 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:00.641523 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:00.641548 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 04:46:00.642342 (+ 794us) client_negotiation.cc:240] Negotiation successful
0925 04:46:00.642437 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1037,"thread_start_us":135,"threads_started":1}
I20260925 04:46:00.643461 11795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f204f2b3ff94d80bef83eade1d7fc66" candidate_uuid: "e253ae9b3a7c4fc6a959087348083508" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94d2139cc2444938aeca87f7baaaf076" is_pre_election: true
I20260925 04:46:00.643836 11795 raft_consensus.cc:2454] T 1f204f2b3ff94d80bef83eade1d7fc66 P 94d2139cc2444938aeca87f7baaaf076 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e253ae9b3a7c4fc6a959087348083508 in term 0.
I20260925 04:46:00.644497 11866 leader_election.cc:302] T 1f204f2b3ff94d80bef83eade1d7fc66 P e253ae9b3a7c4fc6a959087348083508 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94d2139cc2444938aeca87f7baaaf076, e253ae9b3a7c4fc6a959087348083508; no voters:
I20260925 04:46:00.644827 12137 raft_consensus.cc:2781] T 1f204f2b3ff94d80bef83eade1d7fc66 P e253ae9b3a7c4fc6a959087348083508 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:46:00.644898 12137 raft_consensus.cc:482] T 1f204f2b3ff94d80bef83eade1d7fc66 P e253ae9b3a7c4fc6a959087348083508 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:46:00.644936 12137 raft_consensus.cc:3037] T 1f204f2b3ff94d80bef83eade1d7fc66 P e253ae9b3a7c4fc6a959087348083508 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:46:00.646416 12137 raft_consensus.cc:504] T 1f204f2b3ff94d80bef83eade1d7fc66 P e253ae9b3a7c4fc6a959087348083508 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8b7389ce35d43ddb1b48891c3cbe5e0" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 39197 } } peers { permanent_uuid: "94d2139cc2444938aeca87f7baaaf076" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 40579 } } peers { permanent_uuid: "e253ae9b3a7c4fc6a959087348083508" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43179 } }
I20260925 04:46:00.646822 12137 leader_election.cc:288] T 1f204f2b3ff94d80bef83eade1d7fc66 P e253ae9b3a7c4fc6a959087348083508 [CANDIDATE]: Term 1 election: Requested vote from peers a8b7389ce35d43ddb1b48891c3cbe5e0 (127.0.98.67:39197), 94d2139cc2444938aeca87f7baaaf076 (127.0.98.65:40579)
I20260925 04:46:00.647343 11795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f204f2b3ff94d80bef83eade1d7fc66" candidate_uuid: "e253ae9b3a7c4fc6a959087348083508" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94d2139cc2444938aeca87f7baaaf076"
I20260925 04:46:00.647466 11795 raft_consensus.cc:3037] T 1f204f2b3ff94d80bef83eade1d7fc66 P 94d2139cc2444938aeca87f7baaaf076 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:46:00.648631 11795 raft_consensus.cc:2454] T 1f204f2b3ff94d80bef83eade1d7fc66 P 94d2139cc2444938aeca87f7baaaf076 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e253ae9b3a7c4fc6a959087348083508 in term 1.
I20260925 04:46:00.648665 12126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:00.636782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:59919 (local address 127.0.98.67:39197)
0925 04:46:00.636917 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 04:46:00.636922 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:46:00.643202 (+ 6280us) server_negotiation.cc:407] Connection header received
0925 04:46:00.643429 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:46:00.643432 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:46:00.643489 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:46:00.643562 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:46:00.643950 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:00.644675 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:00.647012 (+ 2337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:00.647647 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:00.647751 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:00.647817 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 04:46:00.648444 (+ 627us) server_negotiation.cc:299] Negotiation successful
0925 04:46:00.648524 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 04:46:00.649065 12141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:00.636668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:39197 (local address 127.0.98.66:59919)
0925 04:46:00.643119 (+ 6451us) negotiation.cc:107] Waiting for socket to connect
0925 04:46:00.643135 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:46:00.643284 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:46:00.643715 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:46:00.643718 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:46:00.643737 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:46:00.643818 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:00.643824 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:00.644855 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:00.644858 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:00.646733 (+ 1875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:00.646746 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:00.648174 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:00.648177 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:00.648276 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:00.648313 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 04:46:00.648857 (+ 544us) client_negotiation.cc:240] Negotiation successful
0925 04:46:00.648936 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6386,"thread_start_us":85,"threads_started":1}
I20260925 04:46:00.649102 11866 leader_election.cc:302] T 1f204f2b3ff94d80bef83eade1d7fc66 P e253ae9b3a7c4fc6a959087348083508 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94d2139cc2444938aeca87f7baaaf076, e253ae9b3a7c4fc6a959087348083508; no voters:
I20260925 04:46:00.649451 12137 raft_consensus.cc:2781] T 1f204f2b3ff94d80bef83eade1d7fc66 P e253ae9b3a7c4fc6a959087348083508 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:46:00.649690 12137 raft_consensus.cc:686] T 1f204f2b3ff94d80bef83eade1d7fc66 P e253ae9b3a7c4fc6a959087348083508 [term 1 LEADER]: Becoming Leader. State: Replica: e253ae9b3a7c4fc6a959087348083508, State: Running, Role: LEADER
I20260925 04:46:00.650070 12137 consensus_queue.cc:237] T 1f204f2b3ff94d80bef83eade1d7fc66 P e253ae9b3a7c4fc6a959087348083508 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8b7389ce35d43ddb1b48891c3cbe5e0" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 39197 } } peers { permanent_uuid: "94d2139cc2444938aeca87f7baaaf076" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 40579 } } peers { permanent_uuid: "e253ae9b3a7c4fc6a959087348083508" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43179 } }
I20260925 04:46:00.653681 12051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f204f2b3ff94d80bef83eade1d7fc66" candidate_uuid: "e253ae9b3a7c4fc6a959087348083508" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8b7389ce35d43ddb1b48891c3cbe5e0" is_pre_election: true
I20260925 04:46:00.653702 11628 catalog_manager.cc:5495] T 1f204f2b3ff94d80bef83eade1d7fc66 P e253ae9b3a7c4fc6a959087348083508 reported cstate change: term changed from 0 to 1, leader changed from <none> to e253ae9b3a7c4fc6a959087348083508 (127.0.98.66). New cstate: current_term: 1 leader_uuid: "e253ae9b3a7c4fc6a959087348083508" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8b7389ce35d43ddb1b48891c3cbe5e0" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 39197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94d2139cc2444938aeca87f7baaaf076" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 40579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e253ae9b3a7c4fc6a959087348083508" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43179 } health_report { overall_health: HEALTHY } } }
I20260925 04:46:00.653997 12051 raft_consensus.cc:2454] T 1f204f2b3ff94d80bef83eade1d7fc66 P a8b7389ce35d43ddb1b48891c3cbe5e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e253ae9b3a7c4fc6a959087348083508 in term 0.
I20260925 04:46:00.654374 12064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f204f2b3ff94d80bef83eade1d7fc66" candidate_uuid: "e253ae9b3a7c4fc6a959087348083508" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8b7389ce35d43ddb1b48891c3cbe5e0"
I20260925 04:46:00.654486 12064 raft_consensus.cc:3037] T 1f204f2b3ff94d80bef83eade1d7fc66 P a8b7389ce35d43ddb1b48891c3cbe5e0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:46:00.655824 12064 raft_consensus.cc:2454] T 1f204f2b3ff94d80bef83eade1d7fc66 P a8b7389ce35d43ddb1b48891c3cbe5e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e253ae9b3a7c4fc6a959087348083508 in term 1.
I20260925 04:46:00.701457 12128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:00.695767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42018 (local address 127.0.98.66:43179)
0925 04:46:00.695918 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 04:46:00.695924 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:46:00.695942 (+ 18us) server_negotiation.cc:407] Connection header received
0925 04:46:00.696044 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:46:00.696050 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:46:00.696120 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:46:00.696242 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:46:00.696662 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:00.697543 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:00.698912 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:00.699275 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:00.699405 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:00.699896 (+ 491us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:46:00.700487 (+ 591us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:46:00.700633 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 04:46:00.701242 (+ 609us) server_negotiation.cc:299] Negotiation successful
0925 04:46:00.701306 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 04:46:00.711475 12064 raft_consensus.cc:1261] T 1f204f2b3ff94d80bef83eade1d7fc66 P a8b7389ce35d43ddb1b48891c3cbe5e0 [term 1 FOLLOWER]: Refusing update from remote peer e253ae9b3a7c4fc6a959087348083508: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:46:00.711486 11795 raft_consensus.cc:1261] T 1f204f2b3ff94d80bef83eade1d7fc66 P 94d2139cc2444938aeca87f7baaaf076 [term 1 FOLLOWER]: Refusing update from remote peer e253ae9b3a7c4fc6a959087348083508: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:46:00.712339 12137 consensus_queue.cc:1035] T 1f204f2b3ff94d80bef83eade1d7fc66 P e253ae9b3a7c4fc6a959087348083508 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94d2139cc2444938aeca87f7baaaf076" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 40579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:46:00.718782 12137 consensus_queue.cc:1035] T 1f204f2b3ff94d80bef83eade1d7fc66 P e253ae9b3a7c4fc6a959087348083508 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8b7389ce35d43ddb1b48891c3cbe5e0" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 39197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:46:00.737265 12146 mvcc.cc:205] Tried to move back new op lower bound from 7333116152663891968 to 7333116152426446848. Current Snapshot: MvccSnapshot[applied={T|T < 7333116152663891968}]
I20260925 04:46:00.744951 12149 mvcc.cc:205] Tried to move back new op lower bound from 7333116152663891968 to 7333116152426446848. Current Snapshot: MvccSnapshot[applied={T|T < 7333116152663891968}]
I20260925 04:46:00.745306 12147 mvcc.cc:205] Tried to move back new op lower bound from 7333116152663891968 to 7333116152426446848. Current Snapshot: MvccSnapshot[applied={T|T < 7333116152663891968}]
I20260925 04:46:00.755621 11628 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35342:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:46:00.755893 11628 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35342:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:46:00.758703 11628 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 330d143c128c4151a9c07256b305061d: Sending DeleteTablet for 3 replicas of tablet 1f204f2b3ff94d80bef83eade1d7fc66
I20260925 04:46:00.759843 11912 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f204f2b3ff94d80bef83eade1d7fc66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:46:00 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:42010
I20260925 04:46:00.759958 11775 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f204f2b3ff94d80bef83eade1d7fc66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:46:00 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:57570
I20260925 04:46:00.760835 12046 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f204f2b3ff94d80bef83eade1d7fc66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:46:00 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:38970
I20260925 04:46:00.761080 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 11709
I20260925 04:46:00.765626 12156 tablet_replica.cc:330] T 1f204f2b3ff94d80bef83eade1d7fc66 P e253ae9b3a7c4fc6a959087348083508: stopping tablet replica
I20260925 04:46:00.765903 12157 tablet_replica.cc:330] T 1f204f2b3ff94d80bef83eade1d7fc66 P a8b7389ce35d43ddb1b48891c3cbe5e0: stopping tablet replica
I20260925 04:46:00.766377 12157 raft_consensus.cc:2229] T 1f204f2b3ff94d80bef83eade1d7fc66 P a8b7389ce35d43ddb1b48891c3cbe5e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:46:00.766736 12156 raft_consensus.cc:2229] T 1f204f2b3ff94d80bef83eade1d7fc66 P e253ae9b3a7c4fc6a959087348083508 [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:46:00.767243 12156 raft_consensus.cc:2258] T 1f204f2b3ff94d80bef83eade1d7fc66 P e253ae9b3a7c4fc6a959087348083508 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:46:00.767627 12157 raft_consensus.cc:2258] T 1f204f2b3ff94d80bef83eade1d7fc66 P a8b7389ce35d43ddb1b48891c3cbe5e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:46:00.770784 12156 ts_tablet_manager.cc:1905] T 1f204f2b3ff94d80bef83eade1d7fc66 P e253ae9b3a7c4fc6a959087348083508: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:46:00.772054 12157 ts_tablet_manager.cc:1905] T 1f204f2b3ff94d80bef83eade1d7fc66 P a8b7389ce35d43ddb1b48891c3cbe5e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:46:00.777161 12157 ts_tablet_manager.cc:1918] T 1f204f2b3ff94d80bef83eade1d7fc66 P a8b7389ce35d43ddb1b48891c3cbe5e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:46:00.781558 12156 ts_tablet_manager.cc:1918] T 1f204f2b3ff94d80bef83eade1d7fc66 P e253ae9b3a7c4fc6a959087348083508: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:46:00.786823 12157 log.cc:1196] T 1f204f2b3ff94d80bef83eade1d7fc66 P a8b7389ce35d43ddb1b48891c3cbe5e0: Deleting WAL directory at /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/ts-2/wal/wals/1f204f2b3ff94d80bef83eade1d7fc66
I20260925 04:46:00.787338 12157 ts_tablet_manager.cc:1939] T 1f204f2b3ff94d80bef83eade1d7fc66 P a8b7389ce35d43ddb1b48891c3cbe5e0: Deleting consensus metadata
I20260925 04:46:00.787508 12156 log.cc:1196] T 1f204f2b3ff94d80bef83eade1d7fc66 P e253ae9b3a7c4fc6a959087348083508: Deleting WAL directory at /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790311449873301-393-0/minicluster-data/ts-1/wal/wals/1f204f2b3ff94d80bef83eade1d7fc66
I20260925 04:46:00.787894 12156 ts_tablet_manager.cc:1939] T 1f204f2b3ff94d80bef83eade1d7fc66 P e253ae9b3a7c4fc6a959087348083508: Deleting consensus metadata
I20260925 04:46:00.789454 11614 catalog_manager.cc:4841] TS a8b7389ce35d43ddb1b48891c3cbe5e0 (127.0.98.67:39197): tablet 1f204f2b3ff94d80bef83eade1d7fc66 (table test-table [id=9780a73895494394a6486673eec22bc6]) successfully deleted
W20260925 04:46:00.792135 11613 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.65:40579 (error 108)
I20260925 04:46:00.792893 11615 catalog_manager.cc:4841] TS e253ae9b3a7c4fc6a959087348083508 (127.0.98.66:43179): tablet 1f204f2b3ff94d80bef83eade1d7fc66 (table test-table [id=9780a73895494394a6486673eec22bc6]) successfully deleted
W20260925 04:46:00.793915 12124 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:46:00.793236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:40579 (local address 127.0.0.1:57578)
0925 04:46:00.793385 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 04:46:00.793403 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 04:46:00.793494 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:46:00.793815 (+ 321us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.65:40579: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":50}
I20260925 04:46:00.794082 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 11848
W20260925 04:46:00.796754 11613 catalog_manager.cc:4584] TS 94d2139cc2444938aeca87f7baaaf076 (127.0.98.65:40579): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1f204f2b3ff94d80bef83eade1d7fc66: Network error: Client connection negotiation failed: client connection to 127.0.98.65:40579: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 04:46:00.801381 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 11983
I20260925 04:46:00.808983 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 11596
I20260925 04:46:00.816468 393 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 04:46:01.151690 393 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 04:46:01.151870 393 mini_ranger.cc:66] Stopping Ranger...
I20260925 04:46:01.497095 393 mini_ranger.cc:68] Stopped Ranger
2026-09-25 04:46:01.497 UTC [11182] LOG: received smart shutdown request
2026-09-25 04:46:01.501 UTC [11182] LOG: background worker "logical replication launcher" (PID 11191) exited with exit code 1
2026-09-25 04:46:01.501 UTC [11186] LOG: shutting down
2026-09-25 04:46:01.682 UTC [11182] LOG: database system is shut down
2026-09-25T04:46:01Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (48173 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:46:01 dist-test-slave-kghz krb5kdc[12167](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:46:01 dist-test-slave-kghz krb5kdc[12167](info): set up 2 sockets
Sep 25 04:46:01 dist-test-slave-kghz krb5kdc[12167](info): commencing operation
krb5kdc: starting...
W20260925 04:46:02.842446 393 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/krb5kdc/test-user.keytab.
Sep 25 04:46:02 dist-test-slave-kghz krb5kdc[12167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:46:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:46:02Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/krb5kdc/rangeradmin_127.0.98.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/krb5kdc/rangeradmin_127.0.98.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/krb5kdc/rangerlookup_127.0.98.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/krb5kdc/rangerlookup_127.0.98.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/krb5kdc/HTTP_127.0.98.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/krb5kdc/HTTP_127.0.98.84@KRBTEST.COM.keytab.
I20260925 04:46:03.012957 393 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskHechoP/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/postgres -l logfile start
2026-09-25 04:46:05.667 UTC [12197] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 04:46:05.667 UTC [12197] LOG: listening on IPv4 address "127.0.98.84", port 59211
2026-09-25 04:46:05.672 UTC [12197] LOG: listening on Unix socket "/tmp/.s.PGSQL.59211"
2026-09-25 04:46:05.694 UTC [12200] LOG: database system was shut down at 2026-09-25 04:46:04 UTC
2026-09-25 04:46:05.698 UTC [12197] LOG: database system is ready to accept connections
I20260925 04:46:06.754220 393 mini_postgres.cc:96] Postgres bound to 59211
2026-09-25 04:46:06.760 UTC [12208] FATAL: database "slave" does not exist
127.0.98.84:59211 - accepting connections
I20260925 04:46:06.761269 393 mini_ranger.cc:161] Starting Ranger...
I20260925 04:46:06.776427 393 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 04:46:07.321215 393 mini_ranger.cc:87] Created ranger Postgres database
I20260925 04:46:07.321347 393 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ranger-admin
2026-09-25 04:46:07,592 [I] DB FLAVOR :POSTGRES
2026-09-25 04:46:07,593 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 04:46:07,593 [I] Checking connection..
2026-09-25 04:46:07,593 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHechoP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHechoP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:59211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 04:46:07,831 [I] Checking connection passed.
2026-09-25 04:46:07,831 [I] --------- Verifying version history table ---------
2026-09-25 04:46:07,831 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHechoP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHechoP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:59211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 04:46:08,093 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 04:46:08,094 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHechoP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHechoP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:59211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 04:46:08,324 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 04:46:08,324 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 04:46:08,324 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHechoP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHechoP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:59211/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskHechoP/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 04:46:08.559 UTC [12272] WARNING: there is no transaction in progress
2026-09-25 04:46:08,565 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 04:46:08,565 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHechoP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHechoP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:59211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 04:46:08,797 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 04:46:08,798 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 04:46:08,798 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHechoP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHechoP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:59211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 04:46:09,059 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHechoP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHechoP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:59211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 04:46:09,318 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHechoP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHechoP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:59211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-kghz.c.gcp-upstream.internal','N') ;"
2026-09-25 04:46:09,542 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHechoP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHechoP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:59211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 04:46:09,783 [I] Table x_portal_user does not exist in database ranger
2026-09-25 04:46:09,784 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHechoP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHechoP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:59211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 04:46:10,053 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 04:46:10,053 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHechoP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHechoP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:59211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 04:46:10,313 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHechoP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHechoP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:59211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 04:46:10,567 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 04:46:10,567 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHechoP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHechoP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:59211/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskHechoP/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 04:46:11.586 UTC [12380] WARNING: there is no transaction in progress
2026-09-25 04:46:11.603 UTC [12380] WARNING: there is no transaction in progress
2026-09-25 04:46:11.620 UTC [12380] WARNING: there is no transaction in progress
2026-09-25 04:46:11.637 UTC [12380] WARNING: there is no transaction in progress
2026-09-25 04:46:11.653 UTC [12380] WARNING: there is no transaction in progress
2026-09-25 04:46:11.675 UTC [12380] WARNING: there is no transaction in progress
2026-09-25 04:46:11.735 UTC [12380] WARNING: there is no transaction in progress
2026-09-25 04:46:11.743 UTC [12380] WARNING: there is no transaction in progress
2026-09-25 04:46:11.759 UTC [12380] WARNING: there is no transaction in progress
2026-09-25 04:46:11.771 UTC [12380] WARNING: there is no transaction in progress
2026-09-25 04:46:11.783 UTC [12380] WARNING: there is no transaction in progress
2026-09-25 04:46:11.797 UTC [12380] WARNING: there is no transaction in progress
2026-09-25 04:46:11.808 UTC [12380] WARNING: there is no transaction in progress
2026-09-25 04:46:11.816 UTC [12380] WARNING: there is no transaction in progress
2026-09-25 04:46:11.824 UTC [12380] WARNING: there is no transaction in progress
2026-09-25 04:46:11.832 UTC [12380] WARNING: there is no transaction in progress
2026-09-25 04:46:12,568 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 04:46:12,568 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHechoP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHechoP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:59211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 04:46:12,822 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 04:46:12,823 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHechoP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHechoP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:59211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 04:46:13,077 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 04:46:13,078 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHechoP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHechoP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:59211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 04:46:13,337 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 04:46:13,337 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHechoP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHechoP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:59211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 04:46:13,585 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHechoP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHechoP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:59211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 04:46:13,837 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHechoP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHechoP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:59211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-kghz.c.gcp-upstream.internal';"
2026-09-25 04:46:14,084 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 04:46:14,085 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHechoP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHechoP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:59211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 04:46:14,344 [I] DB_PATCHES have already been applied
I20260925 04:46:14.350425 393 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ranger-admin:/tmp/dist-test-taskHechoP/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskHechoP/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskHechoP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHechoP/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 04:46:14.350486 393 mini_ranger.cc:193] Using host: 127.0.98.84
I20260925 04:46:14.353826 393 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.84:56055
Sep 25, 2026 4:46:14 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 4:46:14 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 4:46:14 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 4:46:14 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 4:46:14 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 4:46:15 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 4:46:15 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 04:46:15.473479 393 mini_ranger.cc:160] Time spent starting Ranger: real 8.712s user 0.000s sys 0.010s
Sep 25, 2026 4:46:35 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 4:46:35 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 25, 2026 4:46:35 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 4:46:35 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 4:46:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 4:46:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 4:46:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 4:46:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 4:46:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 4:46:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 4:46:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 4:46:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 4:46:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 4:46:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 4:46:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 4:46:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 4:46:35 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 4:46:36 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 04:46:37.362437 393 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/krb5kdc/rangerkms_127.0.98.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/krb5kdc/rangerkms_127.0.98.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/krb5kdc/HTTP_127.0.98.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/krb5kdc/HTTP_127.0.98.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 04:46:37 dist-test-slave-kghz krb5kdc[12167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311597, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 04:46:37.450052 393 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 04:46:37.460208 393 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 04:46:38.410123 393 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 04:46:38.410213 393 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ranger-kms
I20260925 04:46:38.410233 393 mini_ranger_kms.cc:227] Using postgres at 127.0.98.84:59211
2026-09-25 04:46:38,581 [I] DB FLAVOR :POSTGRES
2026-09-25 04:46:38,581 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 04:46:38,581 [I] Checking connection
2026-09-25 04:46:38,581 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHechoP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:59211/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 04:46:38,811 [I] connection success
2026-09-25 04:46:38,811 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 04:46:38,812 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 04:46:38,812 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHechoP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:59211/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 04:46:39,048 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 04:46:39,049 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 04:46:39,049 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 04:46:39,049 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHechoP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:59211/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 04:46:39,326 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 04:46:39.795745 393 mini_ranger_kms.cc:310] Created kms service
I20260925 04:46:40.005846 393 mini_ranger_kms.cc:326] Created kudu user
I20260925 04:46:40.054736 393 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 04:46:40 dist-test-slave-kghz krb5kdc[12167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311597, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.84@KRBTEST.COM
I20260925 04:46:40.457331 393 mini_ranger_kms.cc:384] Added ranger policy
I20260925 04:46:40.457494 393 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ranger-kms:/tmp/dist-test-taskHechoP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHechoP/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskHechoP/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 04:46:40.457518 393 mini_ranger_kms.cc:240] Using host: 127.0.98.84
I20260925 04:46:40.461095 393 mini_ranger_kms.cc:276] Ranger KMS PID: 12560
I20260925 04:46:40.461180 393 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.84:35599
Sep 25, 2026 4:46:41 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 4:46:41 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 4:46:41 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 4:46:41 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 4:46:41 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 4:46:41 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260925 04:46:42.668325 393 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.218s user 0.019s sys 0.137s
I20260925 04:46:42.668429 393 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 04:46:42.668506 393 mini_ranger_kms.cc:401] 127.0.98.84:35599/kms/v1/keys
Sep 25 04:46:42 dist-test-slave-kghz krb5kdc[12167](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790311602, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 25 04:46:43 dist-test-slave-kghz krb5kdc[12167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311602, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.84@KRBTEST.COM for HTTP/127.0.98.84@KRBTEST.COM
Sep 25 04:46:43 dist-test-slave-kghz krb5kdc[12167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311602, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.84@KRBTEST.COM for HTTP/127.0.98.84@KRBTEST.COM
Sep 25 04:46:45 dist-test-slave-kghz krb5kdc[12167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260925 04:46:45.793740 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:38837
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:42637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:38837
--ranger_config_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.84:35599/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:46:45.909226 12611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:46:45.909523 12611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:46:45.909567 12611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:46:45.913656 12611 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:46:45.913725 12611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:46:45.913745 12611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:46:45.913761 12611 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:46:45.913777 12611 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:46:45.919584 12611 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:42637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.84:35599/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:38837
--ranger_config_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:38837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12611
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:46:45.920711 12611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:46:45.921772 12611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:46:45.930608 12616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:46:45.930756 12619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:46:45.930608 12617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:46:45.930876 12611 server_base.cc:1034] running on GCE node
I20260925 04:46:45.931528 12611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:46:45.932725 12611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:46:45.933987 12611 hybrid_clock.cc:648] HybridClock initialized: now 1790311605933973 us; error 32 us; skew 500 ppm
Sep 25 04:46:45 dist-test-slave-kghz krb5kdc[12167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311605, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:46:45.937067 12611 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260925 04:46:45.938135 12611 webserver.cc:460] Webserver started at http://127.0.98.126:44031/ using document root <none> and password file <none>
I20260925 04:46:45.938732 12611 fs_manager.cc:359] Metadata directory not provided
I20260925 04:46:45.938812 12611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:46:45.939034 12611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 04:46:45 dist-test-slave-kghz krb5kdc[12167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311605, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for HTTP/127.0.98.84@KRBTEST.COM
I20260925 04:46:46.001287 12611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/master-0/data/instance:
uuid: "304196d4815141698045896c95b84d2b"
format_stamp: "Formatted at 2026-09-25 04:46:45 on dist-test-slave-kghz"
server_key: "37ba36a68fb8f33dc716bfd3df909b32"
server_key_iv: "ef815690f31ece57037bf4ab33b8af05"
server_key_version: "kuduclusterkey@0"
I20260925 04:46:46.001982 12611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/master-0/wal/instance:
uuid: "304196d4815141698045896c95b84d2b"
format_stamp: "Formatted at 2026-09-25 04:46:45 on dist-test-slave-kghz"
server_key: "37ba36a68fb8f33dc716bfd3df909b32"
server_key_iv: "ef815690f31ece57037bf4ab33b8af05"
server_key_version: "kuduclusterkey@0"
I20260925 04:46:46.006152 12611 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 04:46:46.021273 12626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:46:46.022861 12611 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:46:46.023070 12611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/master-0/wal
uuid: "304196d4815141698045896c95b84d2b"
format_stamp: "Formatted at 2026-09-25 04:46:45 on dist-test-slave-kghz"
server_key: "37ba36a68fb8f33dc716bfd3df909b32"
server_key_iv: "ef815690f31ece57037bf4ab33b8af05"
server_key_version: "kuduclusterkey@0"
I20260925 04:46:46.023216 12611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:46:46.037122 12611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:46:46.040378 12611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:46:46.040591 12611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:46:46.048796 12611 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:38837
I20260925 04:46:46.048832 12678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:38837 every 8 connection(s)
I20260925 04:46:46.049799 12611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
I20260925 04:46:46.051698 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 12611
I20260925 04:46:46.051863 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/master-0/wal/instance
I20260925 04:46:46.052147 393 external_mini_cluster.cc:1428] Setting key 1d901c8ca592d917ed3c95f9f5bab118
I20260925 04:46:46.053233 12679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:46:46.060221 12679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 304196d4815141698045896c95b84d2b: Bootstrap starting.
Sep 25 04:46:46 dist-test-slave-kghz krb5kdc[12167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:46:46.062847 12679 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 304196d4815141698045896c95b84d2b: Neither blocks nor log segments found. Creating new log.
I20260925 04:46:46.064169 12679 log.cc:824] T 00000000000000000000000000000000 P 304196d4815141698045896c95b84d2b: Log is configured to *not* fsync() on all Append() calls
I20260925 04:46:46.066406 12679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 304196d4815141698045896c95b84d2b: No bootstrap required, opened a new log
I20260925 04:46:46.066632 12682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:46.053603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59872 (local address 127.0.98.126:38837)
0925 04:46:46.054006 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0925 04:46:46.054026 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 04:46:46.054291 (+ 265us) server_negotiation.cc:407] Connection header received
0925 04:46:46.055017 (+ 726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:46:46.055033 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:46:46.055363 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:46:46.055709 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:46:46.056554 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:46.057570 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:46.058516 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:46.058991 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:46.059173 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:46.061797 (+ 2624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:46:46.061802 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:46:46.061815 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:46:46.061849 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:46:46.063563 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:46:46.064009 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:46:46.064022 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:46:46.064027 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:46:46.064129 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:46:46.064444 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:46:46.064447 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:46:46.064448 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:46:46.065008 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:46:46.065211 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0925 04:46:46.065478 (+ 267us) server_negotiation.cc:299] Negotiation successful
0925 04:46:46.065768 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":125,"threads_started":1}
I20260925 04:46:46.070302 12679 raft_consensus.cc:348] T 00000000000000000000000000000000 P 304196d4815141698045896c95b84d2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "304196d4815141698045896c95b84d2b" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 38837 } }
I20260925 04:46:46.070593 12679 raft_consensus.cc:374] T 00000000000000000000000000000000 P 304196d4815141698045896c95b84d2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:46:46.070652 12679 raft_consensus.cc:729] T 00000000000000000000000000000000 P 304196d4815141698045896c95b84d2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 304196d4815141698045896c95b84d2b, State: Initialized, Role: FOLLOWER
I20260925 04:46:46.071120 12679 consensus_queue.cc:260] T 00000000000000000000000000000000 P 304196d4815141698045896c95b84d2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "304196d4815141698045896c95b84d2b" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 38837 } }
I20260925 04:46:46.071269 12679 raft_consensus.cc:388] T 00000000000000000000000000000000 P 304196d4815141698045896c95b84d2b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:46:46.071331 12679 raft_consensus.cc:482] T 00000000000000000000000000000000 P 304196d4815141698045896c95b84d2b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:46:46.071422 12679 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 304196d4815141698045896c95b84d2b [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:46:46.072575 12679 raft_consensus.cc:504] T 00000000000000000000000000000000 P 304196d4815141698045896c95b84d2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "304196d4815141698045896c95b84d2b" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 38837 } }
I20260925 04:46:46.072942 12679 leader_election.cc:302] T 00000000000000000000000000000000 P 304196d4815141698045896c95b84d2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 304196d4815141698045896c95b84d2b; no voters:
I20260925 04:46:46.073292 12679 leader_election.cc:288] T 00000000000000000000000000000000 P 304196d4815141698045896c95b84d2b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:46:46.073702 12685 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 304196d4815141698045896c95b84d2b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:46:46.074000 12685 raft_consensus.cc:686] T 00000000000000000000000000000000 P 304196d4815141698045896c95b84d2b [term 1 LEADER]: Becoming Leader. State: Replica: 304196d4815141698045896c95b84d2b, State: Running, Role: LEADER
I20260925 04:46:46.074394 12685 consensus_queue.cc:237] T 00000000000000000000000000000000 P 304196d4815141698045896c95b84d2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "304196d4815141698045896c95b84d2b" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 38837 } }
I20260925 04:46:46.074644 12679 sys_catalog.cc:564] T 00000000000000000000000000000000 P 304196d4815141698045896c95b84d2b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:46:46.075631 12685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 304196d4815141698045896c95b84d2b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 304196d4815141698045896c95b84d2b. Latest consensus state: current_term: 1 leader_uuid: "304196d4815141698045896c95b84d2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "304196d4815141698045896c95b84d2b" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 38837 } } }
I20260925 04:46:46.075841 12685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 304196d4815141698045896c95b84d2b [sys.catalog]: This master's current role is: LEADER
I20260925 04:46:46.076020 12686 sys_catalog.cc:455] T 00000000000000000000000000000000 P 304196d4815141698045896c95b84d2b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "304196d4815141698045896c95b84d2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "304196d4815141698045896c95b84d2b" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 38837 } } }
I20260925 04:46:46.076157 12686 sys_catalog.cc:458] T 00000000000000000000000000000000 P 304196d4815141698045896c95b84d2b [sys.catalog]: This master's current role is: LEADER
I20260925 04:46:46.076287 12693 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:46:46.079434 12693 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:46:46.079876 12679 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 04:46:46.086448 12693 catalog_manager.cc:1347] Generated new cluster ID: 202360ad9be54a42ba22ca9b4136f620
I20260925 04:46:46.086506 12693 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:46:46.103976 12693 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:46:46.105338 12693 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:46:46.114722 12693 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 304196d4815141698045896c95b84d2b: Generated new TSK 0
I20260925 04:46:46.115782 12693 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/krb5kdc/kudu_127.0.98.65.keytab.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/krb5kdc/kudu_127.0.98.65.keytab.
I20260925 04:46:46.891785 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.84:35599/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:0
--local_ip_for_outbound_sockets=127.0.98.65
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:38837
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:42637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:46:47.058035 12724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:46:47.058450 12724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:46:47.058557 12724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:46:47.063759 12724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:46:47.063855 12724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:46:47.063961 12724 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260925 04:46:47.069465 12724 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:42637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.84:35599/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:38837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12724
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:46:47.070744 12724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:46:47.072145 12724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:46:47.083307 12729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:46:47.083880 12732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:46:47.088696 12730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:46:47.090070 12724 server_base.cc:1034] running on GCE node
I20260925 04:46:47.090732 12724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:46:47.091581 12724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:46:47.107162 12724 hybrid_clock.cc:648] HybridClock initialized: now 1790311607107095 us; error 77 us; skew 500 ppm
Sep 25 04:46:47 dist-test-slave-kghz krb5kdc[12167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311607, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:46:47.111336 12724 init.cc:377] Logged in from keytab as kudu/127.0.98.65@KRBTEST.COM (short username kudu)
I20260925 04:46:47.112602 12724 webserver.cc:460] Webserver started at http://127.0.98.65:41513/ using document root <none> and password file <none>
I20260925 04:46:47.113345 12724 fs_manager.cc:359] Metadata directory not provided
I20260925 04:46:47.113443 12724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:46:47.113698 12724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 04:46:47 dist-test-slave-kghz krb5kdc[12167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311607, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for HTTP/127.0.98.84@KRBTEST.COM
I20260925 04:46:47.138510 12724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-0/data/instance:
uuid: "9f1fbdb392184927ba779f6dfc43cd32"
format_stamp: "Formatted at 2026-09-25 04:46:47 on dist-test-slave-kghz"
server_key: "0f04488c230faa8959b0ac95a8aa2470"
server_key_iv: "f039ace4db61054953c905c43e6940ea"
server_key_version: "kuduclusterkey@0"
I20260925 04:46:47.139360 12724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-0/wal/instance:
uuid: "9f1fbdb392184927ba779f6dfc43cd32"
format_stamp: "Formatted at 2026-09-25 04:46:47 on dist-test-slave-kghz"
server_key: "0f04488c230faa8959b0ac95a8aa2470"
server_key_iv: "f039ace4db61054953c905c43e6940ea"
server_key_version: "kuduclusterkey@0"
I20260925 04:46:47.143743 12724 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260925 04:46:47.179092 12741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:46:47.180879 12724 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.001s sys 0.000s
I20260925 04:46:47.181075 12724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-0/wal
uuid: "9f1fbdb392184927ba779f6dfc43cd32"
format_stamp: "Formatted at 2026-09-25 04:46:47 on dist-test-slave-kghz"
server_key: "0f04488c230faa8959b0ac95a8aa2470"
server_key_iv: "f039ace4db61054953c905c43e6940ea"
server_key_version: "kuduclusterkey@0"
I20260925 04:46:47.181242 12724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:46:47.201951 12724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:46:47.205765 12724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:46:47.206007 12724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:46:47.206769 12724 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:46:47.207860 12724 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:46:47.207971 12724 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:46:47.208096 12724 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:46:47.208173 12724 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:46:47.220180 12724 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:41313
I20260925 04:46:47.220235 12856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:41313 every 8 connection(s)
I20260925 04:46:47.221546 12724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
I20260925 04:46:47.223424 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 12724
I20260925 04:46:47.223564 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-0/wal/instance
I20260925 04:46:47.223910 393 external_mini_cluster.cc:1428] Setting key 252e62a6092580a3739a86bf82800e5a
WARNING: no policy specified for kudu/127.0.98.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.66@KRBTEST.COM" created.
Sep 25 04:46:47 dist-test-slave-kghz krb5kdc[12167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311607, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:46:47.245712 12859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:47.224360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38837 (local address 127.0.98.65:33659)
0925 04:46:47.225761 (+ 1401us) negotiation.cc:107] Waiting for socket to connect
0925 04:46:47.225816 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 04:46:47.226903 (+ 1087us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:46:47.231297 (+ 4394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:46:47.231307 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:46:47.231673 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:46:47.232366 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:47.232386 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:47.233616 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:47.233620 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:47.235035 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:47.235045 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:47.235717 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:47.235721 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:47.235934 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:47.236665 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:46:47.236687 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:46:47.239997 (+ 3310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:46:47.242394 (+ 2397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:46:47.242401 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:46:47.242420 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:46:47.242760 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:46:47.243087 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:46:47.243091 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:46:47.243093 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:46:47.243264 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:46:47.243711 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:46:47.243719 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:46:47.244073 (+ 354us) client_negotiation.cc:769] Sending connection context
0925 04:46:47.244267 (+ 194us) client_negotiation.cc:240] Negotiation successful
0925 04:46:47.244615 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1174,"thread_start_us":129,"threads_started":1}
I20260925 04:46:47.246824 12860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:47.224540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:33659 (local address 127.0.98.126:38837)
0925 04:46:47.230807 (+ 6267us) server_negotiation.cc:206] Beginning negotiation
0925 04:46:47.230812 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:46:47.230831 (+ 19us) server_negotiation.cc:407] Connection header received
0925 04:46:47.230876 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:46:47.230892 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:46:47.230944 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:46:47.232520 (+ 1576us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:46:47.232635 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:47.233472 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:47.235183 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:47.235539 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:47.236277 (+ 738us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:47.240221 (+ 3944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:46:47.240225 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:46:47.240228 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:46:47.240257 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:46:47.242271 (+ 2014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:46:47.242891 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:46:47.242895 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:46:47.242897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:46:47.242986 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:46:47.243370 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:46:47.243373 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:46:47.243376 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:46:47.243594 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:46:47.246433 (+ 2839us) server_negotiation.cc:1035] Waiting for connection context
0925 04:46:47.246517 (+ 84us) server_negotiation.cc:299] Negotiation successful
0925 04:46:47.246652 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6173,"thread_start_us":68,"threads_started":1}
I20260925 04:46:47.248098 12857 heartbeater.cc:347] Connected to a master server at 127.0.98.126:38837
I20260925 04:46:47.248339 12857 heartbeater.cc:464] Registering TS with master...
I20260925 04:46:47.248903 12857 heartbeater.cc:511] Master 127.0.98.126:38837 requested a full tablet report, sending...
I20260925 04:46:47.250739 12643 ts_manager.cc:194] Registered new tserver with Master: 9f1fbdb392184927ba779f6dfc43cd32 (127.0.98.65:41313)
I20260925 04:46:47.252272 12643 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.65@KRBTEST.COM'} at 127.0.98.65:33659
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/krb5kdc/kudu_127.0.98.66.keytab.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/krb5kdc/kudu_127.0.98.66.keytab.
I20260925 04:46:47.263410 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.84:35599/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:0
--local_ip_for_outbound_sockets=127.0.98.66
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:38837
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:42637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 04:46:47 dist-test-slave-kghz krb5kdc[12167](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790311607, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 04:46:47.420187 12863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:46:47.420631 12863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:46:47.420748 12863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:46:47.426960 12863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:46:47.427043 12863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:46:47.427131 12863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260925 04:46:47.434038 12863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:42637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.84:35599/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:38837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12863
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:46:47.435302 12863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:46:47.436384 12863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:46:47.447868 12869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:46:47.448096 12870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:46:47.447973 12872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:46:47.448359 12863 server_base.cc:1034] running on GCE node
I20260925 04:46:47.448905 12863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:46:47.449818 12863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:46:47.451004 12863 hybrid_clock.cc:648] HybridClock initialized: now 1790311607450969 us; error 42 us; skew 500 ppm
Sep 25 04:46:47 dist-test-slave-kghz krb5kdc[12167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311607, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:46:47.454662 12863 init.cc:377] Logged in from keytab as kudu/127.0.98.66@KRBTEST.COM (short username kudu)
I20260925 04:46:47.455575 12863 webserver.cc:460] Webserver started at http://127.0.98.66:45713/ using document root <none> and password file <none>
I20260925 04:46:47.456164 12863 fs_manager.cc:359] Metadata directory not provided
I20260925 04:46:47.456225 12863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:46:47.456437 12863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 04:46:47 dist-test-slave-kghz krb5kdc[12167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311607, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for HTTP/127.0.98.84@KRBTEST.COM
I20260925 04:46:47.477989 12863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-1/data/instance:
uuid: "6d06553ef26b4efc8953f1e1404011a1"
format_stamp: "Formatted at 2026-09-25 04:46:47 on dist-test-slave-kghz"
server_key: "598c751c151d79a096176a5333f9c156"
server_key_iv: "7ece7a335f9ec8e732cba79979a9b0b7"
server_key_version: "kuduclusterkey@0"
I20260925 04:46:47.479125 12863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-1/wal/instance:
uuid: "6d06553ef26b4efc8953f1e1404011a1"
format_stamp: "Formatted at 2026-09-25 04:46:47 on dist-test-slave-kghz"
server_key: "598c751c151d79a096176a5333f9c156"
server_key_iv: "7ece7a335f9ec8e732cba79979a9b0b7"
server_key_version: "kuduclusterkey@0"
I20260925 04:46:47.485050 12863 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260925 04:46:47.506647 12879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:46:47.508299 12863 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.003s sys 0.001s
I20260925 04:46:47.508456 12863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-1/wal
uuid: "6d06553ef26b4efc8953f1e1404011a1"
format_stamp: "Formatted at 2026-09-25 04:46:47 on dist-test-slave-kghz"
server_key: "598c751c151d79a096176a5333f9c156"
server_key_iv: "7ece7a335f9ec8e732cba79979a9b0b7"
server_key_version: "kuduclusterkey@0"
I20260925 04:46:47.508620 12863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:46:47.538192 12863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:46:47.542054 12863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:46:47.542338 12863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:46:47.543215 12863 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:46:47.544471 12863 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:46:47.544544 12863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:46:47.544615 12863 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:46:47.544655 12863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:46:47.556528 12863 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:35609
I20260925 04:46:47.557796 12863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
I20260925 04:46:47.559958 12992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:35609 every 8 connection(s)
I20260925 04:46:47.563233 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 12863
I20260925 04:46:47.563391 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-1/wal/instance
I20260925 04:46:47.563722 393 external_mini_cluster.cc:1428] Setting key 73a65f363f37538abc3d407919d3eb7c
Sep 25 04:46:47 dist-test-slave-kghz krb5kdc[12167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311607, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:46:47.577750 12860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:47.563275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:53809 (local address 127.0.98.126:38837)
0925 04:46:47.564189 (+ 914us) server_negotiation.cc:206] Beginning negotiation
0925 04:46:47.564194 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:46:47.564214 (+ 20us) server_negotiation.cc:407] Connection header received
0925 04:46:47.564959 (+ 745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:46:47.564963 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:46:47.565014 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:46:47.565063 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:46:47.566755 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:47.567644 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:47.569359 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:47.569724 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:47.569800 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:47.573417 (+ 3617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:46:47.573421 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:46:47.573425 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:46:47.573455 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:46:47.575228 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:46:47.575872 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:46:47.575876 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:46:47.575878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:46:47.575964 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:46:47.576391 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:46:47.576395 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:46:47.576396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:46:47.576619 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:46:47.576707 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 04:46:47.577438 (+ 731us) server_negotiation.cc:299] Negotiation successful
0925 04:46:47.577576 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":819}
I20260925 04:46:47.579602 12995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:47.562287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38837 (local address 127.0.98.66:53809)
0925 04:46:47.563605 (+ 1318us) negotiation.cc:107] Waiting for socket to connect
0925 04:46:47.563664 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 04:46:47.564688 (+ 1024us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:46:47.565467 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:46:47.565476 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:46:47.565836 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:46:47.566581 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:47.566602 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:47.567807 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:47.567810 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:47.569218 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:47.569229 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:47.570012 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:47.570016 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:47.570210 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:47.570944 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:46:47.570965 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:46:47.573223 (+ 2258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:46:47.575368 (+ 2145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:46:47.575374 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:46:47.575391 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:46:47.575748 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:46:47.576092 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:46:47.576095 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:46:47.576096 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:46:47.576275 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:46:47.576776 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:46:47.576784 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:46:47.577127 (+ 343us) client_negotiation.cc:769] Sending connection context
0925 04:46:47.578153 (+ 1026us) client_negotiation.cc:240] Negotiation successful
0925 04:46:47.578536 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1079,"thread_start_us":158,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.67@KRBTEST.COM; defaulting to no policy
I20260925 04:46:47.581175 12993 heartbeater.cc:347] Connected to a master server at 127.0.98.126:38837
I20260925 04:46:47.581465 12993 heartbeater.cc:464] Registering TS with master...
I20260925 04:46:47.582041 12993 heartbeater.cc:511] Master 127.0.98.126:38837 requested a full tablet report, sending...
I20260925 04:46:47.583220 12643 ts_manager.cc:194] Registered new tserver with Master: 6d06553ef26b4efc8953f1e1404011a1 (127.0.98.66:35609)
I20260925 04:46:47.584127 12643 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.66@KRBTEST.COM'} at 127.0.98.66:53809
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/krb5kdc/kudu_127.0.98.67.keytab.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/krb5kdc/kudu_127.0.98.67.keytab.
I20260925 04:46:47.605298 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.84:35599/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:0
--local_ip_for_outbound_sockets=127.0.98.67
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:38837
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:42637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 04:46:47 dist-test-slave-kghz krb5kdc[12167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311607, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for HTTP/127.0.98.84@KRBTEST.COM
W20260925 04:46:47.763675 12998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:46:47.764129 12998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:46:47.764231 12998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:46:47.770296 12998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:46:47.770464 12998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:46:47.770593 12998 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260925 04:46:47.778421 12998 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:42637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.84:35599/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:38837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12998
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:46:47.779942 12998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:46:47.781252 12998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:46:47.794535 13004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:46:47.795045 12998 server_base.cc:1034] running on GCE node
W20260925 04:46:47.795405 13003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:46:47.795719 13006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:46:47.796329 12998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:46:47.797137 12998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:46:47.798334 12998 hybrid_clock.cc:648] HybridClock initialized: now 1790311607798268 us; error 82 us; skew 500 ppm
Sep 25 04:46:47 dist-test-slave-kghz krb5kdc[12167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311607, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:46:47.802027 12998 init.cc:377] Logged in from keytab as kudu/127.0.98.67@KRBTEST.COM (short username kudu)
I20260925 04:46:47.803095 12998 webserver.cc:460] Webserver started at http://127.0.98.67:39893/ using document root <none> and password file <none>
I20260925 04:46:47.803771 12998 fs_manager.cc:359] Metadata directory not provided
I20260925 04:46:47.803833 12998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:46:47.804054 12998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 04:46:47 dist-test-slave-kghz krb5kdc[12167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311607, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for HTTP/127.0.98.84@KRBTEST.COM
I20260925 04:46:47.824747 12998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-2/data/instance:
uuid: "20fb1af143434a7e847b3d840c67baba"
format_stamp: "Formatted at 2026-09-25 04:46:47 on dist-test-slave-kghz"
server_key: "f5b681286238d676a4855710d18b0fbf"
server_key_iv: "497e99a85c4bf4ec8da57f7d7355292d"
server_key_version: "kuduclusterkey@0"
I20260925 04:46:47.826531 12998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-2/wal/instance:
uuid: "20fb1af143434a7e847b3d840c67baba"
format_stamp: "Formatted at 2026-09-25 04:46:47 on dist-test-slave-kghz"
server_key: "f5b681286238d676a4855710d18b0fbf"
server_key_iv: "497e99a85c4bf4ec8da57f7d7355292d"
server_key_version: "kuduclusterkey@0"
I20260925 04:46:47.831761 12998 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 04:46:47.850633 13013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:46:47.852367 12998 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260925 04:46:47.852494 12998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-2/wal
uuid: "20fb1af143434a7e847b3d840c67baba"
format_stamp: "Formatted at 2026-09-25 04:46:47 on dist-test-slave-kghz"
server_key: "f5b681286238d676a4855710d18b0fbf"
server_key_iv: "497e99a85c4bf4ec8da57f7d7355292d"
server_key_version: "kuduclusterkey@0"
I20260925 04:46:47.852620 12998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:46:47.883926 12998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:46:47.887687 12998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:46:47.887933 12998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:46:47.888567 12998 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:46:47.889557 12998 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:46:47.889623 12998 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:46:47.889674 12998 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:46:47.889746 12998 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:46:47.900779 12998 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:36959
I20260925 04:46:47.900848 13126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:36959 every 8 connection(s)
I20260925 04:46:47.901999 12998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
I20260925 04:46:47.904173 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 12998
I20260925 04:46:47.904299 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-2/wal/instance
I20260925 04:46:47.904618 393 external_mini_cluster.cc:1428] Setting key df9cab024812fc5c8eaf7d3afba12595
Sep 25 04:46:47 dist-test-slave-kghz krb5kdc[12167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311607, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:46:47.918772 12860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:47.905995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:44007 (local address 127.0.98.126:38837)
0925 04:46:47.906204 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 04:46:47.906208 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:46:47.906385 (+ 177us) server_negotiation.cc:407] Connection header received
0925 04:46:47.907721 (+ 1336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:46:47.907725 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:46:47.907776 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:46:47.907865 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:46:47.909231 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:47.910051 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:47.911642 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:47.911978 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:47.912095 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:47.914881 (+ 2786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:46:47.914896 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:46:47.914899 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:46:47.914924 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:46:47.916540 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:46:47.917119 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:46:47.917123 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:46:47.917125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:46:47.917201 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:46:47.917569 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:46:47.917572 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:46:47.917574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:46:47.917768 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:46:47.917840 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 04:46:47.918511 (+ 671us) server_negotiation.cc:299] Negotiation successful
0925 04:46:47.918621 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
I20260925 04:46:47.920501 13129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:47.904959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38837 (local address 127.0.98.67:44007)
0925 04:46:47.906206 (+ 1247us) negotiation.cc:107] Waiting for socket to connect
0925 04:46:47.906293 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0925 04:46:47.907476 (+ 1183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:46:47.908077 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:46:47.908087 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:46:47.908420 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:46:47.909064 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:47.909084 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:47.910191 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:47.910195 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:47.911512 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:47.911522 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:47.912136 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:47.912140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:47.912321 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:47.913040 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:46:47.913061 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:46:47.914707 (+ 1646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:46:47.916663 (+ 1956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:46:47.916669 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:46:47.916684 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:46:47.917011 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:46:47.917292 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:46:47.917295 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:46:47.917297 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:46:47.917471 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:46:47.917918 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:46:47.917924 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:46:47.918223 (+ 299us) client_negotiation.cc:769] Sending connection context
0925 04:46:47.919198 (+ 975us) client_negotiation.cc:240] Negotiation successful
0925 04:46:47.919511 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1012,"thread_start_us":133,"threads_started":1}
I20260925 04:46:47.921645 13127 heartbeater.cc:347] Connected to a master server at 127.0.98.126:38837
I20260925 04:46:47.921896 13127 heartbeater.cc:464] Registering TS with master...
I20260925 04:46:47.922453 13127 heartbeater.cc:511] Master 127.0.98.126:38837 requested a full tablet report, sending...
I20260925 04:46:47.923527 12643 ts_manager.cc:194] Registered new tserver with Master: 20fb1af143434a7e847b3d840c67baba (127.0.98.67:36959)
I20260925 04:46:47.924314 12643 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.67@KRBTEST.COM'} at 127.0.98.67:44007
I20260925 04:46:47.931217 393 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 04:46:47.932057 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 12724
I20260925 04:46:47.939385 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 12863
I20260925 04:46:47.946419 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 12998
I20260925 04:46:47.954476 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 12611
I20260925 04:46:47.961346 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:38837
--webserver_interface=127.0.98.126
--webserver_port=44031
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:42637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:38837
--ranger_config_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.84:35599/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.84:35599/kms with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:46:48.072479 13130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:46:48.072746 13130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:46:48.072786 13130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:46:48.077068 13130 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:46:48.077142 13130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:46:48.077162 13130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:46:48.077179 13130 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:46:48.077239 13130 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:46:48.083076 13130 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:42637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.84:35599/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:38837
--ranger_config_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:38837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=44031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13130
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:46:48.084138 13130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:46:48.085238 13130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:46:48.093789 13138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:46:48.093842 13130 server_base.cc:1034] running on GCE node
W20260925 04:46:48.093781 13136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:46:48.093782 13135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:46:48.094673 13130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:46:48.095749 13130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:46:48.096951 13130 hybrid_clock.cc:648] HybridClock initialized: now 1790311608096928 us; error 34 us; skew 500 ppm
Sep 25 04:46:48 dist-test-slave-kghz krb5kdc[12167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311608, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:46:48.099898 13130 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260925 04:46:48.100719 13130 webserver.cc:460] Webserver started at http://127.0.98.126:44031/ using document root <none> and password file <none>
I20260925 04:46:48.101212 13130 fs_manager.cc:359] Metadata directory not provided
I20260925 04:46:48.101274 13130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 04:46:48 dist-test-slave-kghz krb5kdc[12167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311608, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for HTTP/127.0.98.84@KRBTEST.COM
I20260925 04:46:48.148854 13130 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 04:46:48.151679 13146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:46:48.153090 13130 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 04:46:48.153255 13130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/master-0/wal
uuid: "304196d4815141698045896c95b84d2b"
format_stamp: "Formatted at 2026-09-25 04:46:45 on dist-test-slave-kghz"
server_key: "37ba36a68fb8f33dc716bfd3df909b32"
server_key_iv: "ef815690f31ece57037bf4ab33b8af05"
server_key_version: "kuduclusterkey@0"
I20260925 04:46:48.153828 13130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:46:48.194167 13130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:46:48.197302 13130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:46:48.197767 13130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:46:48.204672 13130 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:38837
I20260925 04:46:48.204697 13198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:38837 every 8 connection(s)
I20260925 04:46:48.205716 13130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
I20260925 04:46:48.209276 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 13130
I20260925 04:46:48.209807 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.84:35599/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:41313
--local_ip_for_outbound_sockets=127.0.98.65
--tserver_master_addrs=127.0.98.126:38837
--webserver_port=41513
--webserver_interface=127.0.98.65
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:42637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.84:35599/kms with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 04:46:48.210420 13199 sys_catalog.cc:263] Verifying existing consensus state
I20260925 04:46:48.212503 13199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 304196d4815141698045896c95b84d2b: Bootstrap starting.
I20260925 04:46:48.229185 13199 log.cc:824] T 00000000000000000000000000000000 P 304196d4815141698045896c95b84d2b: Log is configured to *not* fsync() on all Append() calls
I20260925 04:46:48.235554 13199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 304196d4815141698045896c95b84d2b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 04:46:48.235949 13199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 304196d4815141698045896c95b84d2b: Bootstrap complete.
I20260925 04:46:48.240084 13199 raft_consensus.cc:348] T 00000000000000000000000000000000 P 304196d4815141698045896c95b84d2b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "304196d4815141698045896c95b84d2b" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 38837 } }
I20260925 04:46:48.240444 13199 raft_consensus.cc:729] T 00000000000000000000000000000000 P 304196d4815141698045896c95b84d2b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 304196d4815141698045896c95b84d2b, State: Initialized, Role: FOLLOWER
I20260925 04:46:48.240916 13199 consensus_queue.cc:260] T 00000000000000000000000000000000 P 304196d4815141698045896c95b84d2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "304196d4815141698045896c95b84d2b" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 38837 } }
I20260925 04:46:48.241034 13199 raft_consensus.cc:388] T 00000000000000000000000000000000 P 304196d4815141698045896c95b84d2b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:46:48.241122 13199 raft_consensus.cc:482] T 00000000000000000000000000000000 P 304196d4815141698045896c95b84d2b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:46:48.241218 13199 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 304196d4815141698045896c95b84d2b [term 1 FOLLOWER]: Advancing to term 2
I20260925 04:46:48.243099 13199 raft_consensus.cc:504] T 00000000000000000000000000000000 P 304196d4815141698045896c95b84d2b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "304196d4815141698045896c95b84d2b" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 38837 } }
I20260925 04:46:48.243419 13199 leader_election.cc:302] T 00000000000000000000000000000000 P 304196d4815141698045896c95b84d2b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 304196d4815141698045896c95b84d2b; no voters:
I20260925 04:46:48.243724 13199 leader_election.cc:288] T 00000000000000000000000000000000 P 304196d4815141698045896c95b84d2b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 04:46:48.243865 13203 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 304196d4815141698045896c95b84d2b [term 2 FOLLOWER]: Leader election won for term 2
I20260925 04:46:48.244109 13203 raft_consensus.cc:686] T 00000000000000000000000000000000 P 304196d4815141698045896c95b84d2b [term 2 LEADER]: Becoming Leader. State: Replica: 304196d4815141698045896c95b84d2b, State: Running, Role: LEADER
I20260925 04:46:48.244410 13203 consensus_queue.cc:237] T 00000000000000000000000000000000 P 304196d4815141698045896c95b84d2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "304196d4815141698045896c95b84d2b" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 38837 } }
I20260925 04:46:48.244880 13199 sys_catalog.cc:564] T 00000000000000000000000000000000 P 304196d4815141698045896c95b84d2b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:46:48.245769 13204 sys_catalog.cc:455] T 00000000000000000000000000000000 P 304196d4815141698045896c95b84d2b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "304196d4815141698045896c95b84d2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "304196d4815141698045896c95b84d2b" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 38837 } } }
I20260925 04:46:48.245891 13204 sys_catalog.cc:458] T 00000000000000000000000000000000 P 304196d4815141698045896c95b84d2b [sys.catalog]: This master's current role is: LEADER
I20260925 04:46:48.246438 13205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 304196d4815141698045896c95b84d2b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 304196d4815141698045896c95b84d2b. Latest consensus state: current_term: 2 leader_uuid: "304196d4815141698045896c95b84d2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "304196d4815141698045896c95b84d2b" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 38837 } } }
I20260925 04:46:48.246620 13205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 304196d4815141698045896c95b84d2b [sys.catalog]: This master's current role is: LEADER
I20260925 04:46:48.246925 13212 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:46:48.247306 13199 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 04:46:48.253135 13212 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:46:48.254283 13212 catalog_manager.cc:1259] Loaded cluster ID: 202360ad9be54a42ba22ca9b4136f620
I20260925 04:46:48.254344 13212 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:46:48.257321 13212 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:46:48.258162 13212 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 304196d4815141698045896c95b84d2b: Loaded TSK: 0
I20260925 04:46:48.259473 13212 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 04:46:48.351326 13201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:46:48.351670 13201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:46:48.351739 13201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:46:48.357846 13201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:46:48.357928 13201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:46:48.358021 13201 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260925 04:46:48.365896 13201 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:42637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.84:35599/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:41313
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=41513
--tserver_master_addrs=127.0.98.126:38837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:46:48.367291 13201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:46:48.368569 13201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:46:48.380692 13239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:46:48.381291 13236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:46:48.380836 13201 server_base.cc:1034] running on GCE node
W20260925 04:46:48.380719 13237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:46:48.381991 13201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:46:48.382783 13201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:46:48.383982 13201 hybrid_clock.cc:648] HybridClock initialized: now 1790311608383951 us; error 47 us; skew 500 ppm
Sep 25 04:46:48 dist-test-slave-kghz krb5kdc[12167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311608, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:46:48.391499 13201 init.cc:377] Logged in from keytab as kudu/127.0.98.65@KRBTEST.COM (short username kudu)
I20260925 04:46:48.392683 13201 webserver.cc:460] Webserver started at http://127.0.98.65:41513/ using document root <none> and password file <none>
I20260925 04:46:48.393400 13201 fs_manager.cc:359] Metadata directory not provided
I20260925 04:46:48.393508 13201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 04:46:48 dist-test-slave-kghz krb5kdc[12167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311608, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for HTTP/127.0.98.84@KRBTEST.COM
I20260925 04:46:48.423976 13201 fs_manager.cc:709] Time spent opening directory manager: real 0.011s user 0.000s sys 0.003s
I20260925 04:46:48.431120 13247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:46:48.432809 13201 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.003s sys 0.000s
I20260925 04:46:48.432983 13201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-0/wal
uuid: "9f1fbdb392184927ba779f6dfc43cd32"
format_stamp: "Formatted at 2026-09-25 04:46:47 on dist-test-slave-kghz"
server_key: "0f04488c230faa8959b0ac95a8aa2470"
server_key_iv: "f039ace4db61054953c905c43e6940ea"
server_key_version: "kuduclusterkey@0"
I20260925 04:46:48.433754 13201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:46:48.452132 13201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:46:48.455775 13201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:46:48.455968 13201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:46:48.456571 13201 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:46:48.457598 13201 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:46:48.457665 13201 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:46:48.457731 13201 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:46:48.457760 13201 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:46:48.471040 13201 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:41313
I20260925 04:46:48.472183 13201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
I20260925 04:46:48.472455 13360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:41313 every 8 connection(s)
I20260925 04:46:48.479583 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 13201
I20260925 04:46:48.480122 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.84:35599/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:35609
--local_ip_for_outbound_sockets=127.0.98.66
--tserver_master_addrs=127.0.98.126:38837
--webserver_port=45713
--webserver_interface=127.0.98.66
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:42637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.84:35599/kms with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 04:46:48 dist-test-slave-kghz krb5kdc[12167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311608, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:46:48.496249 13364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:48.474872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38837 (local address 127.0.98.65:59777)
0925 04:46:48.475395 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0925 04:46:48.475448 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 04:46:48.476419 (+ 971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:46:48.478399 (+ 1980us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:46:48.478408 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:46:48.478761 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:46:48.481816 (+ 3055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:48.481843 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:48.483907 (+ 2064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:48.483913 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:48.485328 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:48.485341 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:48.486729 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:48.486733 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:48.486929 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:48.487704 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:46:48.487726 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:46:48.489627 (+ 1901us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:46:48.492796 (+ 3169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:46:48.492803 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:46:48.492817 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:46:48.493153 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:46:48.493533 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:46:48.493536 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:46:48.493538 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:46:48.493704 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:46:48.494448 (+ 744us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:46:48.494454 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:46:48.494761 (+ 307us) client_negotiation.cc:769] Sending connection context
0925 04:46:48.494954 (+ 193us) client_negotiation.cc:240] Negotiation successful
0925 04:46:48.495267 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":131,"threads_started":1}
I20260925 04:46:48.498001 13363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:48.474721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:59777 (local address 127.0.98.126:38837)
0925 04:46:48.475963 (+ 1242us) server_negotiation.cc:206] Beginning negotiation
0925 04:46:48.475986 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0925 04:46:48.476036 (+ 50us) server_negotiation.cc:407] Connection header received
0925 04:46:48.477036 (+ 1000us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:46:48.477053 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:46:48.477494 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:46:48.477849 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:46:48.482042 (+ 4193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:48.483737 (+ 1695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:48.485476 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:48.486304 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:48.486472 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:48.489870 (+ 3398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:46:48.489879 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:46:48.489901 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:46:48.489943 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:46:48.492638 (+ 2695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:46:48.493291 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:46:48.493297 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:46:48.493303 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:46:48.493433 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:46:48.493805 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:46:48.493809 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:46:48.493810 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:46:48.494273 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:46:48.494386 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 04:46:48.496852 (+ 2466us) server_negotiation.cc:299] Negotiation successful
0925 04:46:48.497121 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1082,"thread_start_us":122,"threads_started":1}
I20260925 04:46:48.500502 13361 heartbeater.cc:347] Connected to a master server at 127.0.98.126:38837
I20260925 04:46:48.500748 13361 heartbeater.cc:464] Registering TS with master...
I20260925 04:46:48.501307 13361 heartbeater.cc:511] Master 127.0.98.126:38837 requested a full tablet report, sending...
W20260925 04:46:48.503459 13361 heartbeater.cc:644] Failed to heartbeat to 127.0.98.126:38837 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 04:46:48.503592 13361 heartbeater.cc:464] Registering TS with master...
I20260925 04:46:48.503764 13361 heartbeater.cc:511] Master 127.0.98.126:38837 requested a full tablet report, sending...
I20260925 04:46:48.504523 13361 heartbeater.cc:464] Registering TS with master...
I20260925 04:46:48.504690 13361 heartbeater.cc:511] Master 127.0.98.126:38837 requested a full tablet report, sending...
W20260925 04:46:48.505367 13361 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 04:46:48.621424 13365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:46:48.621711 13365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:46:48.621775 13365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:46:48.628176 13365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:46:48.628257 13365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:46:48.628348 13365 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260925 04:46:48.636514 13365 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:42637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.84:35599/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:35609
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=45713
--tserver_master_addrs=127.0.98.126:38837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13365
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:46:48.637969 13365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:46:48.639348 13365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:46:48.652386 13370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:46:48.652388 13371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:46:48.652685 13373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:46:48.652801 13365 server_base.cc:1034] running on GCE node
I20260925 04:46:48.653182 13365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:46:48.653986 13365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:46:48.657334 13365 hybrid_clock.cc:648] HybridClock initialized: now 1790311608657288 us; error 53 us; skew 500 ppm
Sep 25 04:46:48 dist-test-slave-kghz krb5kdc[12167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311608, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:46:48.661154 13365 init.cc:377] Logged in from keytab as kudu/127.0.98.66@KRBTEST.COM (short username kudu)
I20260925 04:46:48.662364 13365 webserver.cc:460] Webserver started at http://127.0.98.66:45713/ using document root <none> and password file <none>
I20260925 04:46:48.662937 13365 fs_manager.cc:359] Metadata directory not provided
I20260925 04:46:48.663057 13365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 04:46:48 dist-test-slave-kghz krb5kdc[12167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311608, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for HTTP/127.0.98.84@KRBTEST.COM
I20260925 04:46:48.684055 13365 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.000s
I20260925 04:46:48.692384 13381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:46:48.694168 13365 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.002s sys 0.000s
I20260925 04:46:48.694383 13365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-1/wal
uuid: "6d06553ef26b4efc8953f1e1404011a1"
format_stamp: "Formatted at 2026-09-25 04:46:47 on dist-test-slave-kghz"
server_key: "598c751c151d79a096176a5333f9c156"
server_key_iv: "7ece7a335f9ec8e732cba79979a9b0b7"
server_key_version: "kuduclusterkey@0"
I20260925 04:46:48.695192 13365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:46:48.727850 13365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:46:48.736443 13365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:46:48.736681 13365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:46:48.737404 13365 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:46:48.738523 13365 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:46:48.738590 13365 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:46:48.738644 13365 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:46:48.738677 13365 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:46:48.752331 13365 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:35609
I20260925 04:46:48.752564 13495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:35609 every 8 connection(s)
I20260925 04:46:48.753756 13365 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
I20260925 04:46:48.759109 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 13365
I20260925 04:46:48.759618 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.84:35599/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:36959
--local_ip_for_outbound_sockets=127.0.98.67
--tserver_master_addrs=127.0.98.126:38837
--webserver_port=39893
--webserver_interface=127.0.98.67
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:42637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.84:35599/kms with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 04:46:48 dist-test-slave-kghz krb5kdc[12167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311608, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:46:48.771428 13363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:48.756698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:44347 (local address 127.0.98.126:38837)
0925 04:46:48.756851 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 04:46:48.756856 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:46:48.758032 (+ 1176us) server_negotiation.cc:407] Connection header received
0925 04:46:48.759324 (+ 1292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:46:48.759328 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:46:48.759388 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:46:48.759442 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:46:48.761058 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:48.761884 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:48.763681 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:48.764035 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:48.764129 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:48.767271 (+ 3142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:46:48.767275 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:46:48.767278 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:46:48.767317 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:46:48.768880 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:46:48.769554 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:46:48.769558 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:46:48.769560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:46:48.769655 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:46:48.770091 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:46:48.770095 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:46:48.770097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:46:48.770372 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:46:48.770465 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 04:46:48.771107 (+ 642us) server_negotiation.cc:299] Negotiation successful
0925 04:46:48.771250 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 04:46:48.773228 13498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:48.757199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38837 (local address 127.0.98.66:44347)
0925 04:46:48.757861 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0925 04:46:48.757923 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0925 04:46:48.759067 (+ 1144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:46:48.759861 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:46:48.759870 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:46:48.760215 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:46:48.760881 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:48.760900 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:48.762074 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:48.762077 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:48.763527 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:48.763538 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:48.764346 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:48.764350 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:48.764560 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:48.765304 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:46:48.765326 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:46:48.767083 (+ 1757us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:46:48.769010 (+ 1927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:46:48.769018 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:46:48.769036 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:46:48.769407 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:46:48.769769 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:46:48.769772 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:46:48.769775 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:46:48.769968 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:46:48.770540 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:46:48.770545 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:46:48.770817 (+ 272us) client_negotiation.cc:769] Sending connection context
0925 04:46:48.771709 (+ 892us) client_negotiation.cc:240] Negotiation successful
0925 04:46:48.772088 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":184,"threads_started":1}
I20260925 04:46:48.774724 13496 heartbeater.cc:347] Connected to a master server at 127.0.98.126:38837
I20260925 04:46:48.775002 13496 heartbeater.cc:464] Registering TS with master...
I20260925 04:46:48.775681 13496 heartbeater.cc:511] Master 127.0.98.126:38837 requested a full tablet report, sending...
W20260925 04:46:48.777539 13496 heartbeater.cc:644] Failed to heartbeat to 127.0.98.126:38837 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 04:46:48.777673 13496 heartbeater.cc:464] Registering TS with master...
I20260925 04:46:48.777864 13496 heartbeater.cc:511] Master 127.0.98.126:38837 requested a full tablet report, sending...
I20260925 04:46:48.778801 13496 heartbeater.cc:464] Registering TS with master...
I20260925 04:46:48.779001 13496 heartbeater.cc:511] Master 127.0.98.126:38837 requested a full tablet report, sending...
W20260925 04:46:48.780198 13496 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 04:46:48.914541 13499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:46:48.914863 13499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:46:48.914947 13499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:46:48.919409 13499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:46:48.919490 13499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:46:48.919600 13499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260925 04:46:48.925768 13499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:42637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.84:35599/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:36959
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=39893
--tserver_master_addrs=127.0.98.126:38837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:46:48.926977 13499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:46:48.928035 13499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:46:48.940356 13505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:46:48.940819 13504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:46:48.940392 13507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:46:48.941182 13499 server_base.cc:1034] running on GCE node
I20260925 04:46:48.941676 13499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:46:48.942521 13499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:46:48.958065 13499 hybrid_clock.cc:648] HybridClock initialized: now 1790311608957987 us; error 77 us; skew 500 ppm
Sep 25 04:46:48 dist-test-slave-kghz krb5kdc[12167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311608, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:46:48.962430 13499 init.cc:377] Logged in from keytab as kudu/127.0.98.67@KRBTEST.COM (short username kudu)
I20260925 04:46:48.963802 13499 webserver.cc:460] Webserver started at http://127.0.98.67:39893/ using document root <none> and password file <none>
I20260925 04:46:48.964663 13499 fs_manager.cc:359] Metadata directory not provided
I20260925 04:46:48.964854 13499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 04:46:48 dist-test-slave-kghz krb5kdc[12167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311608, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for HTTP/127.0.98.84@KRBTEST.COM
I20260925 04:46:48.988585 13499 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.003s
I20260925 04:46:49.001741 13515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:46:49.003810 13499 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.003s sys 0.001s
I20260925 04:46:49.004037 13499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-2/wal
uuid: "20fb1af143434a7e847b3d840c67baba"
format_stamp: "Formatted at 2026-09-25 04:46:47 on dist-test-slave-kghz"
server_key: "f5b681286238d676a4855710d18b0fbf"
server_key_iv: "497e99a85c4bf4ec8da57f7d7355292d"
server_key_version: "kuduclusterkey@0"
I20260925 04:46:49.004948 13499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:46:49.029176 13499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:46:49.033075 13499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:46:49.033283 13499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:46:49.034055 13499 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:46:49.035168 13499 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:46:49.035223 13499 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:46:49.035279 13499 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:46:49.035306 13499 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:46:49.047230 13499 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:36959
I20260925 04:46:49.047300 13635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:36959 every 8 connection(s)
I20260925 04:46:49.048520 13499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
I20260925 04:46:49.058092 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 13499
Sep 25 04:46:49 dist-test-slave-kghz krb5kdc[12167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311608, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:46:49.066953 13363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:49.051265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:46597 (local address 127.0.98.126:38837)
0925 04:46:49.051413 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 04:46:49.051417 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:46:49.052476 (+ 1059us) server_negotiation.cc:407] Connection header received
0925 04:46:49.053688 (+ 1212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:46:49.053692 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:46:49.053757 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:46:49.053858 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:46:49.055244 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:49.056080 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:49.057738 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:49.058103 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:49.058214 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:49.061369 (+ 3155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:46:49.061373 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:46:49.061376 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:46:49.061403 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:46:49.063269 (+ 1866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:46:49.063891 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:46:49.063895 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:46:49.063897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:46:49.063983 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:46:49.065003 (+ 1020us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:46:49.065007 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:46:49.065009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:46:49.065239 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:46:49.065314 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 04:46:49.066645 (+ 1331us) server_negotiation.cc:299] Negotiation successful
0925 04:46:49.066780 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 04:46:49.068602 13638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:49.051757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38837 (local address 127.0.98.67:46597)
0925 04:46:49.052311 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0925 04:46:49.052369 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 04:46:49.053416 (+ 1047us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:46:49.054072 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:46:49.054081 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:46:49.054447 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:46:49.055073 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:49.055091 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:49.056227 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:49.056231 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:49.057611 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:49.057622 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:49.058484 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:49.058490 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:49.058711 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:49.059376 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:46:49.059397 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:46:49.061189 (+ 1792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:46:49.063398 (+ 2209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:46:49.063406 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:46:49.063423 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:46:49.063773 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:46:49.064111 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:46:49.064114 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:46:49.064116 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:46:49.064293 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:46:49.065951 (+ 1658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:46:49.065958 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:46:49.066351 (+ 393us) client_negotiation.cc:769] Sending connection context
0925 04:46:49.067193 (+ 842us) client_negotiation.cc:240] Negotiation successful
0925 04:46:49.067530 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":141,"threads_started":1}
I20260925 04:46:49.070972 13636 heartbeater.cc:347] Connected to a master server at 127.0.98.126:38837
I20260925 04:46:49.071237 13636 heartbeater.cc:464] Registering TS with master...
I20260925 04:46:49.071818 13636 heartbeater.cc:511] Master 127.0.98.126:38837 requested a full tablet report, sending...
I20260925 04:46:49.073388 13159 ts_manager.cc:194] Registered new tserver with Master: 20fb1af143434a7e847b3d840c67baba (127.0.98.67:36959)
I20260925 04:46:49.075052 13159 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.67@KRBTEST.COM'} at 127.0.98.67:46597
I20260925 04:46:49.075982 13640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:49.059409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59882 (local address 127.0.98.126:38837)
0925 04:46:49.059654 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0925 04:46:49.059662 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:46:49.060052 (+ 390us) server_negotiation.cc:407] Connection header received
0925 04:46:49.060092 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:46:49.060094 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:46:49.060139 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:46:49.060222 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:46:49.061372 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:49.062818 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:49.065348 (+ 2530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:49.065694 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:49.065802 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:49.068997 (+ 3195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:46:49.069001 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:46:49.069003 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:46:49.069039 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:46:49.070743 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:46:49.073001 (+ 2258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:46:49.073005 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:46:49.073007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:46:49.073109 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:46:49.074569 (+ 1460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:46:49.074573 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:46:49.074575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:46:49.074774 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:46:49.074946 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0925 04:46:49.075690 (+ 744us) server_negotiation.cc:299] Negotiation successful
0925 04:46:49.075811 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":88,"threads_started":1}
Sep 25 04:46:49 dist-test-slave-kghz krb5kdc[12167](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790311609, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:46:49.505612 13361 heartbeater.cc:464] Registering TS with master...
I20260925 04:46:49.505949 13361 heartbeater.cc:511] Master 127.0.98.126:38837 requested a full tablet report, sending...
I20260925 04:46:49.506832 13159 ts_manager.cc:194] Registered new tserver with Master: 9f1fbdb392184927ba779f6dfc43cd32 (127.0.98.65:41313)
I20260925 04:46:49.507642 13159 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.65@KRBTEST.COM'} at 127.0.98.65:59777
Sep 25 04:46:49 dist-test-slave-kghz krb5kdc[12167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311609, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for HTTP/127.0.98.84@KRBTEST.COM
I20260925 04:46:49.780460 13496 heartbeater.cc:464] Registering TS with master...
I20260925 04:46:49.780771 13496 heartbeater.cc:511] Master 127.0.98.126:38837 requested a full tablet report, sending...
I20260925 04:46:49.781775 13163 ts_manager.cc:194] Registered new tserver with Master: 6d06553ef26b4efc8953f1e1404011a1 (127.0.98.66:35609)
I20260925 04:46:49.782596 13163 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.66@KRBTEST.COM'} at 127.0.98.66:44347
I20260925 04:46:49.786481 393 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 04:46:49.800040 13650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:49.788767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46498 (local address 127.0.98.126:38837)
0925 04:46:49.789384 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0925 04:46:49.789389 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:46:49.789777 (+ 388us) server_negotiation.cc:407] Connection header received
0925 04:46:49.789829 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:46:49.789833 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:46:49.789889 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:46:49.789980 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:46:49.791018 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:49.791863 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:49.793004 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:49.793359 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:49.793472 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:49.796384 (+ 2912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:46:49.796388 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:46:49.796391 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:46:49.796419 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:46:49.798207 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:46:49.798731 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:46:49.798735 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:46:49.798737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:46:49.798825 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:46:49.799133 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:46:49.799136 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:46:49.799138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:46:49.799341 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:46:49.799466 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 04:46:49.799744 (+ 278us) server_negotiation.cc:299] Negotiation successful
0925 04:46:49.799867 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":89,"threads_started":1}
I20260925 04:46:49.804754 13163 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46498:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:46:49.809887 13163 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:46:49.837546 13658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:49.827157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:36959 (local address 127.0.0.1:38128)
0925 04:46:49.829906 (+ 2749us) negotiation.cc:107] Waiting for socket to connect
0925 04:46:49.829918 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 04:46:49.829998 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:46:49.830939 (+ 941us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:46:49.830946 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:46:49.830968 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:46:49.831107 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:49.831114 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:49.833368 (+ 2254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:49.833372 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:49.834850 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:49.834858 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:49.836937 (+ 2079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:49.836941 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:49.837060 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:49.837101 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 04:46:49.837219 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 04:46:49.837345 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2686,"thread_start_us":90,"threads_started":1}
I20260925 04:46:49.838186 13659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:49.827568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38128 (local address 127.0.98.67:36959)
0925 04:46:49.830235 (+ 2667us) server_negotiation.cc:206] Beginning negotiation
0925 04:46:49.830265 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0925 04:46:49.830289 (+ 24us) server_negotiation.cc:407] Connection header received
0925 04:46:49.830367 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:46:49.830372 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:46:49.830612 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:46:49.830761 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:46:49.831243 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:49.833224 (+ 1981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:49.834979 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:49.836613 (+ 1634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:49.836714 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:49.836876 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 04:46:49.837938 (+ 1062us) server_negotiation.cc:299] Negotiation successful
0925 04:46:49.838048 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2557,"thread_start_us":111,"threads_started":1}
I20260925 04:46:49.841501 13563 tablet_service.cc:1467] Processing CreateTablet for tablet af8f38768c2c4fdbb81b4d953c2e5c1c (DEFAULT_TABLE table=test-table [id=4d59c6cee57f4713bfa8a5c567d01bdf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:46:49.842060 13660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:49.825941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:41313 (local address 127.0.0.1:45644)
0925 04:46:49.829486 (+ 3545us) negotiation.cc:107] Waiting for socket to connect
0925 04:46:49.829540 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 04:46:49.829672 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:46:49.835398 (+ 5726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:46:49.835401 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:46:49.835419 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:46:49.835521 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:49.835527 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:49.839095 (+ 3568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:49.839099 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:49.840177 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:49.840189 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:49.841625 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:49.841629 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:49.841719 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:49.841744 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 04:46:49.841843 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 04:46:49.841927 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3350,"thread_start_us":117,"threads_started":1}
I20260925 04:46:49.842815 13563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af8f38768c2c4fdbb81b4d953c2e5c1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:46:49.845253 13656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:49.828301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:35609 (local address 127.0.0.1:41476)
0925 04:46:49.834400 (+ 6099us) negotiation.cc:107] Waiting for socket to connect
0925 04:46:49.834415 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:46:49.834502 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:46:49.840895 (+ 6393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:46:49.840899 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:46:49.840933 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:46:49.841022 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:49.841028 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:49.842645 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:49.842649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:49.843837 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:49.843848 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:49.844888 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:49.844892 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:49.844975 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:49.844999 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 04:46:49.845056 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 04:46:49.845118 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6030,"thread_start_us":108,"threads_started":1}
I20260925 04:46:49.846812 13661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:49.828452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41476 (local address 127.0.98.66:35609)
0925 04:46:49.838613 (+ 10161us) server_negotiation.cc:206] Beginning negotiation
0925 04:46:49.838621 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:46:49.838645 (+ 24us) server_negotiation.cc:407] Connection header received
0925 04:46:49.838718 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:46:49.838723 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:46:49.838925 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:46:49.840939 (+ 2014us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:46:49.841138 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:49.842514 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:49.843995 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:49.844756 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:49.845588 (+ 832us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:49.846545 (+ 957us) server_negotiation.cc:1035] Waiting for connection context
0925 04:46:49.846596 (+ 51us) server_negotiation.cc:299] Negotiation successful
0925 04:46:49.846687 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10047,"thread_start_us":72,"threads_started":1}
I20260925 04:46:49.847602 13657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:49.826058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45644 (local address 127.0.98.65:41313)
0925 04:46:49.834743 (+ 8685us) server_negotiation.cc:206] Beginning negotiation
0925 04:46:49.834750 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:46:49.834774 (+ 24us) server_negotiation.cc:407] Connection header received
0925 04:46:49.834847 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:46:49.834853 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:46:49.835087 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:46:49.835234 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:46:49.835658 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:49.836737 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:49.840341 (+ 3604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:49.841446 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:49.847183 (+ 5737us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:49.847328 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 04:46:49.847376 (+ 48us) server_negotiation.cc:299] Negotiation successful
0925 04:46:49.847465 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8576,"thread_start_us":92,"threads_started":1}
I20260925 04:46:49.849876 13420 tablet_service.cc:1467] Processing CreateTablet for tablet af8f38768c2c4fdbb81b4d953c2e5c1c (DEFAULT_TABLE table=test-table [id=4d59c6cee57f4713bfa8a5c567d01bdf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:46:49.850963 13420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af8f38768c2c4fdbb81b4d953c2e5c1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:46:49.859010 13295 tablet_service.cc:1467] Processing CreateTablet for tablet af8f38768c2c4fdbb81b4d953c2e5c1c (DEFAULT_TABLE table=test-table [id=4d59c6cee57f4713bfa8a5c567d01bdf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:46:49.860262 13295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af8f38768c2c4fdbb81b4d953c2e5c1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:46:49.862666 13662 tablet_bootstrap.cc:492] T af8f38768c2c4fdbb81b4d953c2e5c1c P 20fb1af143434a7e847b3d840c67baba: Bootstrap starting.
I20260925 04:46:49.869566 13662 tablet_bootstrap.cc:654] T af8f38768c2c4fdbb81b4d953c2e5c1c P 20fb1af143434a7e847b3d840c67baba: Neither blocks nor log segments found. Creating new log.
I20260925 04:46:49.870425 13662 log.cc:824] T af8f38768c2c4fdbb81b4d953c2e5c1c P 20fb1af143434a7e847b3d840c67baba: Log is configured to *not* fsync() on all Append() calls
I20260925 04:46:49.870740 13663 tablet_bootstrap.cc:492] T af8f38768c2c4fdbb81b4d953c2e5c1c P 6d06553ef26b4efc8953f1e1404011a1: Bootstrap starting.
I20260925 04:46:49.873558 13663 tablet_bootstrap.cc:654] T af8f38768c2c4fdbb81b4d953c2e5c1c P 6d06553ef26b4efc8953f1e1404011a1: Neither blocks nor log segments found. Creating new log.
I20260925 04:46:49.874637 13663 log.cc:824] T af8f38768c2c4fdbb81b4d953c2e5c1c P 6d06553ef26b4efc8953f1e1404011a1: Log is configured to *not* fsync() on all Append() calls
I20260925 04:46:49.877013 13662 tablet_bootstrap.cc:492] T af8f38768c2c4fdbb81b4d953c2e5c1c P 20fb1af143434a7e847b3d840c67baba: No bootstrap required, opened a new log
I20260925 04:46:49.877287 13662 ts_tablet_manager.cc:1397] T af8f38768c2c4fdbb81b4d953c2e5c1c P 20fb1af143434a7e847b3d840c67baba: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
I20260925 04:46:49.877619 13663 tablet_bootstrap.cc:492] T af8f38768c2c4fdbb81b4d953c2e5c1c P 6d06553ef26b4efc8953f1e1404011a1: No bootstrap required, opened a new log
I20260925 04:46:49.877864 13663 ts_tablet_manager.cc:1397] T af8f38768c2c4fdbb81b4d953c2e5c1c P 6d06553ef26b4efc8953f1e1404011a1: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260925 04:46:49.882560 13663 raft_consensus.cc:348] T af8f38768c2c4fdbb81b4d953c2e5c1c P 6d06553ef26b4efc8953f1e1404011a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1fbdb392184927ba779f6dfc43cd32" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41313 } } peers { permanent_uuid: "20fb1af143434a7e847b3d840c67baba" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 36959 } } peers { permanent_uuid: "6d06553ef26b4efc8953f1e1404011a1" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 35609 } }
I20260925 04:46:49.882560 13662 raft_consensus.cc:348] T af8f38768c2c4fdbb81b4d953c2e5c1c P 20fb1af143434a7e847b3d840c67baba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1fbdb392184927ba779f6dfc43cd32" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41313 } } peers { permanent_uuid: "20fb1af143434a7e847b3d840c67baba" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 36959 } } peers { permanent_uuid: "6d06553ef26b4efc8953f1e1404011a1" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 35609 } }
I20260925 04:46:49.882879 13662 raft_consensus.cc:374] T af8f38768c2c4fdbb81b4d953c2e5c1c P 20fb1af143434a7e847b3d840c67baba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:46:49.882936 13662 raft_consensus.cc:729] T af8f38768c2c4fdbb81b4d953c2e5c1c P 20fb1af143434a7e847b3d840c67baba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20fb1af143434a7e847b3d840c67baba, State: Initialized, Role: FOLLOWER
I20260925 04:46:49.883162 13663 raft_consensus.cc:374] T af8f38768c2c4fdbb81b4d953c2e5c1c P 6d06553ef26b4efc8953f1e1404011a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:46:49.883275 13663 raft_consensus.cc:729] T af8f38768c2c4fdbb81b4d953c2e5c1c P 6d06553ef26b4efc8953f1e1404011a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d06553ef26b4efc8953f1e1404011a1, State: Initialized, Role: FOLLOWER
I20260925 04:46:49.883472 13662 consensus_queue.cc:260] T af8f38768c2c4fdbb81b4d953c2e5c1c P 20fb1af143434a7e847b3d840c67baba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1fbdb392184927ba779f6dfc43cd32" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41313 } } peers { permanent_uuid: "20fb1af143434a7e847b3d840c67baba" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 36959 } } peers { permanent_uuid: "6d06553ef26b4efc8953f1e1404011a1" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 35609 } }
I20260925 04:46:49.883903 13663 consensus_queue.cc:260] T af8f38768c2c4fdbb81b4d953c2e5c1c P 6d06553ef26b4efc8953f1e1404011a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1fbdb392184927ba779f6dfc43cd32" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41313 } } peers { permanent_uuid: "20fb1af143434a7e847b3d840c67baba" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 36959 } } peers { permanent_uuid: "6d06553ef26b4efc8953f1e1404011a1" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 35609 } }
I20260925 04:46:49.884634 13636 heartbeater.cc:503] Master 127.0.98.126:38837 was elected leader, sending a full tablet report...
I20260925 04:46:49.885152 13664 tablet_bootstrap.cc:492] T af8f38768c2c4fdbb81b4d953c2e5c1c P 9f1fbdb392184927ba779f6dfc43cd32: Bootstrap starting.
I20260925 04:46:49.885644 13663 ts_tablet_manager.cc:1428] T af8f38768c2c4fdbb81b4d953c2e5c1c P 6d06553ef26b4efc8953f1e1404011a1: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260925 04:46:49.886008 13496 heartbeater.cc:503] Master 127.0.98.126:38837 was elected leader, sending a full tablet report...
I20260925 04:46:49.886312 13662 ts_tablet_manager.cc:1428] T af8f38768c2c4fdbb81b4d953c2e5c1c P 20fb1af143434a7e847b3d840c67baba: Time spent starting tablet: real 0.009s user 0.009s sys 0.000s
I20260925 04:46:49.888286 13664 tablet_bootstrap.cc:654] T af8f38768c2c4fdbb81b4d953c2e5c1c P 9f1fbdb392184927ba779f6dfc43cd32: Neither blocks nor log segments found. Creating new log.
I20260925 04:46:49.889257 13664 log.cc:824] T af8f38768c2c4fdbb81b4d953c2e5c1c P 9f1fbdb392184927ba779f6dfc43cd32: Log is configured to *not* fsync() on all Append() calls
I20260925 04:46:49.898880 13664 tablet_bootstrap.cc:492] T af8f38768c2c4fdbb81b4d953c2e5c1c P 9f1fbdb392184927ba779f6dfc43cd32: No bootstrap required, opened a new log
I20260925 04:46:49.899180 13664 ts_tablet_manager.cc:1397] T af8f38768c2c4fdbb81b4d953c2e5c1c P 9f1fbdb392184927ba779f6dfc43cd32: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.000s
I20260925 04:46:49.903381 13664 raft_consensus.cc:348] T af8f38768c2c4fdbb81b4d953c2e5c1c P 9f1fbdb392184927ba779f6dfc43cd32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1fbdb392184927ba779f6dfc43cd32" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41313 } } peers { permanent_uuid: "20fb1af143434a7e847b3d840c67baba" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 36959 } } peers { permanent_uuid: "6d06553ef26b4efc8953f1e1404011a1" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 35609 } }
I20260925 04:46:49.903792 13664 raft_consensus.cc:374] T af8f38768c2c4fdbb81b4d953c2e5c1c P 9f1fbdb392184927ba779f6dfc43cd32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:46:49.903873 13664 raft_consensus.cc:729] T af8f38768c2c4fdbb81b4d953c2e5c1c P 9f1fbdb392184927ba779f6dfc43cd32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f1fbdb392184927ba779f6dfc43cd32, State: Initialized, Role: FOLLOWER
I20260925 04:46:49.904439 13664 consensus_queue.cc:260] T af8f38768c2c4fdbb81b4d953c2e5c1c P 9f1fbdb392184927ba779f6dfc43cd32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1fbdb392184927ba779f6dfc43cd32" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41313 } } peers { permanent_uuid: "20fb1af143434a7e847b3d840c67baba" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 36959 } } peers { permanent_uuid: "6d06553ef26b4efc8953f1e1404011a1" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 35609 } }
I20260925 04:46:49.905555 13664 ts_tablet_manager.cc:1428] T af8f38768c2c4fdbb81b4d953c2e5c1c P 9f1fbdb392184927ba779f6dfc43cd32: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260925 04:46:49.905969 13361 heartbeater.cc:503] Master 127.0.98.126:38837 was elected leader, sending a full tablet report...
W20260925 04:46:49.974340 13362 tablet.cc:2369] T af8f38768c2c4fdbb81b4d953c2e5c1c P 9f1fbdb392184927ba779f6dfc43cd32: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 04:46:50.005824 13497 tablet.cc:2369] T af8f38768c2c4fdbb81b4d953c2e5c1c P 6d06553ef26b4efc8953f1e1404011a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 04:46:50 dist-test-slave-kghz krb5kdc[12167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311609, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for HTTP/127.0.98.84@KRBTEST.COM
W20260925 04:46:50.050431 13637 tablet.cc:2369] T af8f38768c2c4fdbb81b4d953c2e5c1c P 20fb1af143434a7e847b3d840c67baba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:46:50.237093 13670 raft_consensus.cc:482] T af8f38768c2c4fdbb81b4d953c2e5c1c P 9f1fbdb392184927ba779f6dfc43cd32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:46:50.237385 13670 raft_consensus.cc:504] T af8f38768c2c4fdbb81b4d953c2e5c1c P 9f1fbdb392184927ba779f6dfc43cd32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1fbdb392184927ba779f6dfc43cd32" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41313 } } peers { permanent_uuid: "20fb1af143434a7e847b3d840c67baba" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 36959 } } peers { permanent_uuid: "6d06553ef26b4efc8953f1e1404011a1" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 35609 } }
I20260925 04:46:50.238938 13670 leader_election.cc:288] T af8f38768c2c4fdbb81b4d953c2e5c1c P 9f1fbdb392184927ba779f6dfc43cd32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20fb1af143434a7e847b3d840c67baba (127.0.98.67:36959), 6d06553ef26b4efc8953f1e1404011a1 (127.0.98.66:35609)
I20260925 04:46:50.248102 13659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:50.240144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:35191 (local address 127.0.98.67:36959)
0925 04:46:50.240304 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 04:46:50.240310 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:46:50.243079 (+ 2769us) server_negotiation.cc:407] Connection header received
0925 04:46:50.243317 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:46:50.243321 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:46:50.243378 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:46:50.243436 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:46:50.243978 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:50.244792 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:50.246696 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:50.247321 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:50.247421 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:50.247490 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 04:46:50.247873 (+ 383us) server_negotiation.cc:299] Negotiation successful
0925 04:46:50.247968 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 04:46:50.248541 13668 raft_consensus.cc:482] T af8f38768c2c4fdbb81b4d953c2e5c1c P 6d06553ef26b4efc8953f1e1404011a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:46:50.248747 13668 raft_consensus.cc:504] T af8f38768c2c4fdbb81b4d953c2e5c1c P 6d06553ef26b4efc8953f1e1404011a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1fbdb392184927ba779f6dfc43cd32" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41313 } } peers { permanent_uuid: "20fb1af143434a7e847b3d840c67baba" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 36959 } } peers { permanent_uuid: "6d06553ef26b4efc8953f1e1404011a1" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 35609 } }
I20260925 04:46:50.248972 13673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:50.239148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:36959 (local address 127.0.98.65:35191)
0925 04:46:50.242925 (+ 3777us) negotiation.cc:107] Waiting for socket to connect
0925 04:46:50.242983 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 04:46:50.243173 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:46:50.243597 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:46:50.243600 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:46:50.243621 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:46:50.243774 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:50.243781 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:50.244998 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:50.245001 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:50.246514 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:50.246556 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:50.247585 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:50.247588 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:50.247725 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:50.247746 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 04:46:50.248533 (+ 787us) client_negotiation.cc:240] Negotiation successful
0925 04:46:50.248610 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3681,"mutex_wait_us":14,"thread_start_us":119,"threads_started":1}
I20260925 04:46:50.249811 13567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af8f38768c2c4fdbb81b4d953c2e5c1c" candidate_uuid: "9f1fbdb392184927ba779f6dfc43cd32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20fb1af143434a7e847b3d840c67baba" is_pre_election: true
I20260925 04:46:50.250125 13567 raft_consensus.cc:2454] T af8f38768c2c4fdbb81b4d953c2e5c1c P 20fb1af143434a7e847b3d840c67baba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f1fbdb392184927ba779f6dfc43cd32 in term 0.
I20260925 04:46:50.250185 13668 leader_election.cc:288] T af8f38768c2c4fdbb81b4d953c2e5c1c P 6d06553ef26b4efc8953f1e1404011a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f1fbdb392184927ba779f6dfc43cd32 (127.0.98.65:41313), 20fb1af143434a7e847b3d840c67baba (127.0.98.67:36959)
I20260925 04:46:50.250945 13251 leader_election.cc:302] T af8f38768c2c4fdbb81b4d953c2e5c1c P 9f1fbdb392184927ba779f6dfc43cd32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20fb1af143434a7e847b3d840c67baba, 9f1fbdb392184927ba779f6dfc43cd32; no voters:
I20260925 04:46:50.251482 13670 raft_consensus.cc:2781] T af8f38768c2c4fdbb81b4d953c2e5c1c P 9f1fbdb392184927ba779f6dfc43cd32 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:46:50.251572 13670 raft_consensus.cc:482] T af8f38768c2c4fdbb81b4d953c2e5c1c P 9f1fbdb392184927ba779f6dfc43cd32 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:46:50.251613 13670 raft_consensus.cc:3037] T af8f38768c2c4fdbb81b4d953c2e5c1c P 9f1fbdb392184927ba779f6dfc43cd32 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:46:50.252306 13661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:50.240315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:55931 (local address 127.0.98.66:35609)
0925 04:46:50.240442 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 04:46:50.240447 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:46:50.244877 (+ 4430us) server_negotiation.cc:407] Connection header received
0925 04:46:50.245063 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:46:50.245067 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:46:50.245117 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:46:50.245161 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:46:50.245532 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:50.246322 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:50.250997 (+ 4675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:50.251629 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:50.251711 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:50.251772 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 04:46:50.252111 (+ 339us) server_negotiation.cc:299] Negotiation successful
0925 04:46:50.252191 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 04:46:50.252718 13674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:50.239595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:35609 (local address 127.0.98.65:55931)
0925 04:46:50.244778 (+ 5183us) negotiation.cc:107] Waiting for socket to connect
0925 04:46:50.244793 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:46:50.244946 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:46:50.245278 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:46:50.245280 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:46:50.245293 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:46:50.245408 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:50.245413 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:50.246538 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:50.246541 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:50.250851 (+ 4310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:50.250864 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:50.251867 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:50.251871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:50.251971 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:50.251992 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 04:46:50.252487 (+ 495us) client_negotiation.cc:240] Negotiation successful
0925 04:46:50.252565 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5114,"mutex_wait_us":3135,"thread_start_us":101,"threads_started":1}
I20260925 04:46:50.253017 13670 raft_consensus.cc:504] T af8f38768c2c4fdbb81b4d953c2e5c1c P 9f1fbdb392184927ba779f6dfc43cd32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1fbdb392184927ba779f6dfc43cd32" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41313 } } peers { permanent_uuid: "20fb1af143434a7e847b3d840c67baba" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 36959 } } peers { permanent_uuid: "6d06553ef26b4efc8953f1e1404011a1" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 35609 } }
I20260925 04:46:50.253398 13670 leader_election.cc:288] T af8f38768c2c4fdbb81b4d953c2e5c1c P 9f1fbdb392184927ba779f6dfc43cd32 [CANDIDATE]: Term 1 election: Requested vote from peers 20fb1af143434a7e847b3d840c67baba (127.0.98.67:36959), 6d06553ef26b4efc8953f1e1404011a1 (127.0.98.66:35609)
I20260925 04:46:50.254619 13446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af8f38768c2c4fdbb81b4d953c2e5c1c" candidate_uuid: "9f1fbdb392184927ba779f6dfc43cd32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d06553ef26b4efc8953f1e1404011a1" is_pre_election: true
I20260925 04:46:50.254837 13446 raft_consensus.cc:2454] T af8f38768c2c4fdbb81b4d953c2e5c1c P 6d06553ef26b4efc8953f1e1404011a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f1fbdb392184927ba779f6dfc43cd32 in term 0.
I20260925 04:46:50.255157 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af8f38768c2c4fdbb81b4d953c2e5c1c" candidate_uuid: "9f1fbdb392184927ba779f6dfc43cd32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d06553ef26b4efc8953f1e1404011a1"
I20260925 04:46:50.255249 13567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af8f38768c2c4fdbb81b4d953c2e5c1c" candidate_uuid: "9f1fbdb392184927ba779f6dfc43cd32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20fb1af143434a7e847b3d840c67baba"
I20260925 04:46:50.255268 13445 raft_consensus.cc:3037] T af8f38768c2c4fdbb81b4d953c2e5c1c P 6d06553ef26b4efc8953f1e1404011a1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:46:50.255370 13567 raft_consensus.cc:3037] T af8f38768c2c4fdbb81b4d953c2e5c1c P 20fb1af143434a7e847b3d840c67baba [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:46:50.256601 13445 raft_consensus.cc:2454] T af8f38768c2c4fdbb81b4d953c2e5c1c P 6d06553ef26b4efc8953f1e1404011a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f1fbdb392184927ba779f6dfc43cd32 in term 1.
I20260925 04:46:50.256623 13567 raft_consensus.cc:2454] T af8f38768c2c4fdbb81b4d953c2e5c1c P 20fb1af143434a7e847b3d840c67baba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f1fbdb392184927ba779f6dfc43cd32 in term 1.
I20260925 04:46:50.257292 13249 leader_election.cc:302] T af8f38768c2c4fdbb81b4d953c2e5c1c P 9f1fbdb392184927ba779f6dfc43cd32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d06553ef26b4efc8953f1e1404011a1, 9f1fbdb392184927ba779f6dfc43cd32; no voters:
I20260925 04:46:50.257507 13670 raft_consensus.cc:2781] T af8f38768c2c4fdbb81b4d953c2e5c1c P 9f1fbdb392184927ba779f6dfc43cd32 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:46:50.257731 13670 raft_consensus.cc:686] T af8f38768c2c4fdbb81b4d953c2e5c1c P 9f1fbdb392184927ba779f6dfc43cd32 [term 1 LEADER]: Becoming Leader. State: Replica: 9f1fbdb392184927ba779f6dfc43cd32, State: Running, Role: LEADER
I20260925 04:46:50.258083 13670 consensus_queue.cc:237] T af8f38768c2c4fdbb81b4d953c2e5c1c P 9f1fbdb392184927ba779f6dfc43cd32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1fbdb392184927ba779f6dfc43cd32" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41313 } } peers { permanent_uuid: "20fb1af143434a7e847b3d840c67baba" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 36959 } } peers { permanent_uuid: "6d06553ef26b4efc8953f1e1404011a1" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 35609 } }
I20260925 04:46:50.262488 13163 catalog_manager.cc:5495] T af8f38768c2c4fdbb81b4d953c2e5c1c P 9f1fbdb392184927ba779f6dfc43cd32 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f1fbdb392184927ba779f6dfc43cd32 (127.0.98.65). New cstate: current_term: 1 leader_uuid: "9f1fbdb392184927ba779f6dfc43cd32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1fbdb392184927ba779f6dfc43cd32" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20fb1af143434a7e847b3d840c67baba" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 36959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d06553ef26b4efc8953f1e1404011a1" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 35609 } health_report { overall_health: UNKNOWN } } }
I20260925 04:46:50.264892 13659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:50.250712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:45491 (local address 127.0.98.67:36959)
0925 04:46:50.257329 (+ 6617us) server_negotiation.cc:206] Beginning negotiation
0925 04:46:50.257334 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:46:50.257347 (+ 13us) server_negotiation.cc:407] Connection header received
0925 04:46:50.257413 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:46:50.257417 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:46:50.257463 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:46:50.257544 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:46:50.258578 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:50.259326 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:50.260831 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:50.261417 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:50.261500 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:50.261564 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 04:46:50.264666 (+ 3102us) server_negotiation.cc:299] Negotiation successful
0925 04:46:50.264760 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6530}
I20260925 04:46:50.265489 13675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:50.250518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:36959 (local address 127.0.98.66:45491)
0925 04:46:50.257131 (+ 6613us) negotiation.cc:107] Waiting for socket to connect
0925 04:46:50.257147 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:46:50.257266 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:46:50.258186 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:46:50.258190 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:46:50.258319 (+ 129us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:46:50.258442 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:50.258450 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:50.259470 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:50.259473 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:50.260693 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:50.260704 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:50.264389 (+ 3685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:50.264395 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:50.264503 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:50.264535 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 04:46:50.265234 (+ 699us) client_negotiation.cc:240] Negotiation successful
0925 04:46:50.265327 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6534,"thread_start_us":110,"threads_started":1}
I20260925 04:46:50.266460 13567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af8f38768c2c4fdbb81b4d953c2e5c1c" candidate_uuid: "6d06553ef26b4efc8953f1e1404011a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20fb1af143434a7e847b3d840c67baba" is_pre_election: true
I20260925 04:46:50.266664 13567 raft_consensus.cc:2379] T af8f38768c2c4fdbb81b4d953c2e5c1c P 20fb1af143434a7e847b3d840c67baba [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d06553ef26b4efc8953f1e1404011a1 in current term 1: Already voted for candidate 9f1fbdb392184927ba779f6dfc43cd32 in this term.
I20260925 04:46:50.271771 13657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:50.250453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:46471 (local address 127.0.98.65:41313)
0925 04:46:50.250635 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 04:46:50.250639 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:46:50.267521 (+ 16882us) server_negotiation.cc:407] Connection header received
0925 04:46:50.267695 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:46:50.267698 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:46:50.267746 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:46:50.267791 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:46:50.268163 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:50.268892 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:50.270462 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:50.271070 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:50.271151 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:50.271220 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 04:46:50.271556 (+ 336us) server_negotiation.cc:299] Negotiation successful
0925 04:46:50.271639 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260925 04:46:50.272396 13675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:50.254589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:41313 (local address 127.0.98.66:46471)
0925 04:46:50.267447 (+ 12858us) negotiation.cc:107] Waiting for socket to connect
0925 04:46:50.267463 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:46:50.267587 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:46:50.267942 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:46:50.267945 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:46:50.267962 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:46:50.268042 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:50.268047 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:50.269051 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:50.269054 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:50.270315 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:50.270326 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:50.271316 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:50.271319 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:50.271406 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:50.271432 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 04:46:50.272186 (+ 754us) client_negotiation.cc:240] Negotiation successful
0925 04:46:50.272268 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12795,"thread_start_us":125,"threads_started":1}
I20260925 04:46:50.273288 13315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af8f38768c2c4fdbb81b4d953c2e5c1c" candidate_uuid: "6d06553ef26b4efc8953f1e1404011a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f1fbdb392184927ba779f6dfc43cd32" is_pre_election: true
I20260925 04:46:50.274108 13382 leader_election.cc:302] T af8f38768c2c4fdbb81b4d953c2e5c1c P 6d06553ef26b4efc8953f1e1404011a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d06553ef26b4efc8953f1e1404011a1; no voters: 20fb1af143434a7e847b3d840c67baba, 9f1fbdb392184927ba779f6dfc43cd32
I20260925 04:46:50.274678 13668 raft_consensus.cc:2726] T af8f38768c2c4fdbb81b4d953c2e5c1c P 6d06553ef26b4efc8953f1e1404011a1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 04:46:50.289126 13657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:50.284006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45660 (local address 127.0.98.65:41313)
0925 04:46:50.284125 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0925 04:46:50.284130 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:46:50.284142 (+ 12us) server_negotiation.cc:407] Connection header received
0925 04:46:50.284225 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:46:50.284228 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:46:50.284292 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:46:50.284394 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:46:50.284777 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:50.285547 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:50.286869 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:50.287226 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:50.287335 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:50.287773 (+ 438us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:46:50.288344 (+ 571us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:46:50.288466 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 04:46:50.288888 (+ 422us) server_negotiation.cc:299] Negotiation successful
0925 04:46:50.288963 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 04:46:50.297662 13445 raft_consensus.cc:1261] T af8f38768c2c4fdbb81b4d953c2e5c1c P 6d06553ef26b4efc8953f1e1404011a1 [term 1 FOLLOWER]: Refusing update from remote peer 9f1fbdb392184927ba779f6dfc43cd32: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:46:50.298342 13677 consensus_queue.cc:1035] T af8f38768c2c4fdbb81b4d953c2e5c1c P 9f1fbdb392184927ba779f6dfc43cd32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d06553ef26b4efc8953f1e1404011a1" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 35609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:46:50.305583 13567 raft_consensus.cc:1261] T af8f38768c2c4fdbb81b4d953c2e5c1c P 20fb1af143434a7e847b3d840c67baba [term 1 FOLLOWER]: Refusing update from remote peer 9f1fbdb392184927ba779f6dfc43cd32: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:46:50.306129 13677 consensus_queue.cc:1035] T af8f38768c2c4fdbb81b4d953c2e5c1c P 9f1fbdb392184927ba779f6dfc43cd32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20fb1af143434a7e847b3d840c67baba" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 36959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:46:50.316457 13680 mvcc.cc:205] Tried to move back new op lower bound from 7333116355770687488 to 7333116355620495360. Current Snapshot: MvccSnapshot[applied={T|T < 7333116355770687488}]
I20260925 04:46:50.321867 13681 mvcc.cc:205] Tried to move back new op lower bound from 7333116355770687488 to 7333116355620495360. Current Snapshot: MvccSnapshot[applied={T|T < 7333116355770687488}]
I20260925 04:46:50.323993 13684 mvcc.cc:205] Tried to move back new op lower bound from 7333116355770687488 to 7333116355620495360. Current Snapshot: MvccSnapshot[applied={T|T < 7333116355770687488}]
I20260925 04:46:50.328382 13163 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46498:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:46:50.328577 13163 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46498:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:46:50.331171 13163 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 304196d4815141698045896c95b84d2b: Sending DeleteTablet for 3 replicas of tablet af8f38768c2c4fdbb81b4d953c2e5c1c
I20260925 04:46:50.332845 13295 tablet_service.cc:1514] Processing DeleteTablet for tablet af8f38768c2c4fdbb81b4d953c2e5c1c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:46:50 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:45644
I20260925 04:46:50.332842 13420 tablet_service.cc:1514] Processing DeleteTablet for tablet af8f38768c2c4fdbb81b4d953c2e5c1c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:46:50 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:41476
I20260925 04:46:50.333357 13563 tablet_service.cc:1514] Processing DeleteTablet for tablet af8f38768c2c4fdbb81b4d953c2e5c1c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:46:50 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:38128
I20260925 04:46:50.334743 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 13201
I20260925 04:46:50.335566 13693 tablet_replica.cc:330] T af8f38768c2c4fdbb81b4d953c2e5c1c P 20fb1af143434a7e847b3d840c67baba: stopping tablet replica
I20260925 04:46:50.335911 13693 raft_consensus.cc:2229] T af8f38768c2c4fdbb81b4d953c2e5c1c P 20fb1af143434a7e847b3d840c67baba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:46:50.336287 13693 raft_consensus.cc:2258] T af8f38768c2c4fdbb81b4d953c2e5c1c P 20fb1af143434a7e847b3d840c67baba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:46:50.336687 13692 tablet_replica.cc:330] T af8f38768c2c4fdbb81b4d953c2e5c1c P 6d06553ef26b4efc8953f1e1404011a1: stopping tablet replica
I20260925 04:46:50.336982 13692 raft_consensus.cc:2229] T af8f38768c2c4fdbb81b4d953c2e5c1c P 6d06553ef26b4efc8953f1e1404011a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:46:50.337268 13692 raft_consensus.cc:2258] T af8f38768c2c4fdbb81b4d953c2e5c1c P 6d06553ef26b4efc8953f1e1404011a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:46:50.339002 13693 ts_tablet_manager.cc:1905] T af8f38768c2c4fdbb81b4d953c2e5c1c P 20fb1af143434a7e847b3d840c67baba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:46:50.341861 13693 ts_tablet_manager.cc:1918] T af8f38768c2c4fdbb81b4d953c2e5c1c P 20fb1af143434a7e847b3d840c67baba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:46:50.341965 13693 log.cc:1196] T af8f38768c2c4fdbb81b4d953c2e5c1c P 20fb1af143434a7e847b3d840c67baba: Deleting WAL directory at /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790311449873301-393-0/minicluster-data/ts-2/wal/wals/af8f38768c2c4fdbb81b4d953c2e5c1c
I20260925 04:46:50.342326 13693 ts_tablet_manager.cc:1939] T af8f38768c2c4fdbb81b4d953c2e5c1c P 20fb1af143434a7e847b3d840c67baba: Deleting consensus metadata
I20260925 04:46:50.343693 13150 catalog_manager.cc:4841] TS 20fb1af143434a7e847b3d840c67baba (127.0.98.67:36959): tablet af8f38768c2c4fdbb81b4d953c2e5c1c (table test-table [id=4d59c6cee57f4713bfa8a5c567d01bdf]) successfully deleted
W20260925 04:46:50.346887 13147 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.65:41313 (error 108)
I20260925 04:46:50.347703 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 13365
I20260925 04:46:50.355099 13700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:50.350986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:41313 (local address 127.0.0.1:45668)
0925 04:46:50.354886 (+ 3900us) negotiation.cc:107] Waiting for socket to connect
0925 04:46:50.354976 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.65:41313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3800,"thread_start_us":58,"threads_started":1}
I20260925 04:46:50.355417 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 13499
W20260925 04:46:50.355561 13147 catalog_manager.cc:4584] TS 9f1fbdb392184927ba779f6dfc43cd32 (127.0.98.65:41313): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet af8f38768c2c4fdbb81b4d953c2e5c1c: Network error: Client connection negotiation failed: client connection to 127.0.98.65:41313: connect: Connection refused (error 111)
I20260925 04:46:50.356650 13700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:50.355243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:35609 (local address 127.0.0.1:41488)
0925 04:46:50.356531 (+ 1288us) negotiation.cc:107] Waiting for socket to connect
0925 04:46:50.356582 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.66:35609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1202}
I20260925 04:46:50.363154 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 13130
I20260925 04:46:50.370025 393 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 04:46:50.715157 393 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 04:46:50.715303 393 mini_ranger.cc:66] Stopping Ranger...
I20260925 04:46:51.060745 393 mini_ranger.cc:68] Stopped Ranger
2026-09-25 04:46:51.060 UTC [12197] LOG: received smart shutdown request
2026-09-25 04:46:51.064 UTC [12197] LOG: background worker "logical replication launcher" (PID 12206) exited with exit code 1
2026-09-25 04:46:51.065 UTC [12201] LOG: shutting down
2026-09-25 04:46:51.250 UTC [12197] LOG: database system is shut down
2026-09-25T04:46:51Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (49569 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311449873301-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:46:51 dist-test-slave-kghz krb5kdc[13709](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:46:51 dist-test-slave-kghz krb5kdc[13709](info): set up 2 sockets
Sep 25 04:46:51 dist-test-slave-kghz krb5kdc[13709](info): commencing operation
krb5kdc: starting...
W20260925 04:46:52.414485 393 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311449873301-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311449873301-393-0/krb5kdc/test-user.keytab.
Sep 25 04:46:52 dist-test-slave-kghz krb5kdc[13709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311612, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:46:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:46:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260925 04:46:52.537818 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311449873301-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311449873301-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311449873301-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:33029
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:45537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:33029,127.0.98.125:46289,127.0.98.124:43861
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:46:52.649487 13722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:46:52.649773 13722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:46:52.649816 13722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:46:52.653884 13722 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:46:52.653951 13722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:46:52.653971 13722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:46:52.653988 13722 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:46:52.654003 13722 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:46:52.659406 13722 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:45537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311449873301-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311449873301-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:33029,127.0.98.125:46289,127.0.98.124:43861
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:33029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311449873301-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:46:52.660492 13722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:46:52.661535 13722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:46:52.670082 13730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:46:52.670082 13727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:46:52.670224 13728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:46:52.670476 13722 server_base.cc:1034] running on GCE node
I20260925 04:46:52.671079 13722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:46:52.672116 13722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:46:52.673388 13722 hybrid_clock.cc:648] HybridClock initialized: now 1790311612673363 us; error 43 us; skew 500 ppm
Sep 25 04:46:52 dist-test-slave-kghz krb5kdc[13709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311612, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:46:52.676337 13722 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260925 04:46:52.677260 13722 webserver.cc:460] Webserver started at http://127.0.98.126:40249/ using document root <none> and password file <none>
I20260925 04:46:52.677834 13722 fs_manager.cc:359] Metadata directory not provided
I20260925 04:46:52.677942 13722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:46:52.678172 13722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:46:52.680203 13722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311449873301-393-0/minicluster-data/master-0/data/instance:
uuid: "aa6306245adf42af962e1a467e6724af"
format_stamp: "Formatted at 2026-09-25 04:46:52 on dist-test-slave-kghz"
server_key: "46e63ac7f619f0b88cf7b0c857f98910"
server_key_iv: "00230935a7843d0295bf53f217e9eefc"
server_key_version: "encryptionkey@0"
I20260925 04:46:52.680764 13722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311449873301-393-0/minicluster-data/master-0/wal/instance:
uuid: "aa6306245adf42af962e1a467e6724af"
format_stamp: "Formatted at 2026-09-25 04:46:52 on dist-test-slave-kghz"
server_key: "46e63ac7f619f0b88cf7b0c857f98910"
server_key_iv: "00230935a7843d0295bf53f217e9eefc"
server_key_version: "encryptionkey@0"
I20260925 04:46:52.684562 13722 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260925 04:46:52.687069 13737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:46:52.688257 13722 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 04:46:52.688400 13722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311449873301-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311449873301-393-0/minicluster-data/master-0/wal
uuid: "aa6306245adf42af962e1a467e6724af"
format_stamp: "Formatted at 2026-09-25 04:46:52 on dist-test-slave-kghz"
server_key: "46e63ac7f619f0b88cf7b0c857f98910"
server_key_iv: "00230935a7843d0295bf53f217e9eefc"
server_key_version: "encryptionkey@0"
I20260925 04:46:52.688534 13722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311449873301-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311449873301-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311449873301-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:46:52.712224 13722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:46:52.715644 13722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:46:52.715853 13722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:46:52.722839 13722 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:33029
I20260925 04:46:52.722851 13789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:33029 every 8 connection(s)
I20260925 04:46:52.723788 13722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
I20260925 04:46:52.724375 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 13722
I20260925 04:46:52.724550 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311449873301-393-0/minicluster-data/master-0/wal/instance
I20260925 04:46:52.724813 393 external_mini_cluster.cc:1428] Setting key 6ccc10eddc33da92a6dd9ae27dd3a33a
I20260925 04:46:52.726986 13790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:46:52.732614 13790 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33029 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.125@KRBTEST.COM" created.
Sep 25 04:46:52 dist-test-slave-kghz krb5kdc[13709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311612, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:46:52.750438 13794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:52.736257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50804 (local address 127.0.98.126:33029)
0925 04:46:52.736557 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0925 04:46:52.736572 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 04:46:52.736622 (+ 50us) server_negotiation.cc:407] Connection header received
0925 04:46:52.737627 (+ 1005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:46:52.737635 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:46:52.737881 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:46:52.738098 (+ 217us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:46:52.739306 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:52.740214 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:52.741894 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:52.742360 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:52.742494 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:52.745234 (+ 2740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:46:52.745239 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:46:52.745249 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:46:52.745269 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:46:52.747074 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:46:52.747588 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:46:52.747594 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:46:52.747595 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:46:52.747702 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:46:52.748021 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:46:52.748026 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:46:52.748027 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:46:52.748514 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:46:52.748656 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 04:46:52.749074 (+ 418us) server_negotiation.cc:299] Negotiation successful
0925 04:46:52.749332 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":140,"threads_started":1}
I20260925 04:46:52.750435 13793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:52.735426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:33029 (local address 127.0.0.1:50804)
0925 04:46:52.735999 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0925 04:46:52.736043 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 04:46:52.737260 (+ 1217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:46:52.738128 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:46:52.738136 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:46:52.738729 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:46:52.739125 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:52.739146 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:52.740355 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:52.740361 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:52.741740 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:52.741756 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:52.742531 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:52.742538 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:52.742709 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:52.743342 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:46:52.743369 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:46:52.745069 (+ 1700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:46:52.747197 (+ 2128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:46:52.747207 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:46:52.747217 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:46:52.747445 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:46:52.747801 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:46:52.747804 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:46:52.747806 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:46:52.747926 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:46:52.748640 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:46:52.748646 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:46:52.748831 (+ 185us) client_negotiation.cc:769] Sending connection context
0925 04:46:52.749015 (+ 184us) client_negotiation.cc:240] Negotiation successful
0925 04:46:52.749407 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"mutex_wait_us":134,"thread_start_us":150,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311449873301-393-0/krb5kdc/kudu_127.0.98.125.keytab.
Entry for principal kudu/127.0.98.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311449873301-393-0/krb5kdc/kudu_127.0.98.125.keytab.
I20260925 04:46:52.753057 13790 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.125" port: 46289 } has no permanent_uuid. Determining permanent_uuid...
I20260925 04:46:52.753816 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311449873301-393-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311449873301-393-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311449873301-393-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311449873301-393-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.125:46289
--webserver_interface=127.0.98.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311449873301-393-0/krb5kdc/kudu_127.0.98.125.keytab
--principal=kudu/127.0.98.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:45537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:33029,127.0.98.125:46289,127.0.98.124:43861
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 04:46:52.753892 13793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:52.753609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:46289 (local address 127.0.0.1:37948)
0925 04:46:52.753760 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 04:46:52.753830 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.125:46289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260925 04:46:52.754124 13740 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.125:46289: connect: Connection refused (error 111)
I20260925 04:46:52.755018 13793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:52.754807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:46289 (local address 127.0.0.1:37954)
0925 04:46:52.754919 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0925 04:46:52.754968 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.125:46289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 04:46:52.755146 13790 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.125:46289: Network error: Client connection negotiation failed: client connection to 127.0.98.125:46289: connect: Connection refused (error 111)
I20260925 04:46:52.804487 13790 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.125" port: 46289 } attempt: 1
I20260925 04:46:52.805500 13793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:52.805166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:46289 (local address 127.0.0.1:37962)
0925 04:46:52.805353 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 04:46:52.805420 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.125:46289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260925 04:46:52.806313 13793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:52.806057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:46289 (local address 127.0.0.1:37976)
0925 04:46:52.806159 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0925 04:46:52.806215 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.125:46289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260925 04:46:52.806459 13790 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.125:46289: Network error: Client connection negotiation failed: client connection to 127.0.98.125:46289: connect: Connection refused (error 111)
W20260925 04:46:52.866966 13797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:46:52.867268 13797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:46:52.867358 13797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:46:52.871639 13797 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:46:52.871737 13797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:46:52.871778 13797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:46:52.871838 13797 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:46:52.871881 13797 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:46:52.874622 13790 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.125" port: 46289 } attempt: 2
I20260925 04:46:52.875586 13793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:52.875262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:46289 (local address 127.0.0.1:37984)
0925 04:46:52.875402 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 04:46:52.875509 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.125:46289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260925 04:46:52.876302 13793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:52.876089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:46289 (local address 127.0.0.1:37990)
0925 04:46:52.876193 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0925 04:46:52.876247 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.125:46289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260925 04:46:52.876430 13790 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.125:46289: Network error: Client connection negotiation failed: client connection to 127.0.98.125:46289: connect: Connection refused (error 111)
I20260925 04:46:52.878050 13797 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:45537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311449873301-393-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311449873301-393-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:33029,127.0.98.125:46289,127.0.98.124:43861
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311449873301-393-0/krb5kdc/kudu_127.0.98.125.keytab
--principal=kudu/127.0.98.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.125:46289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311449873301-393-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311449873301-393-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:46:52.879315 13797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:46:52.880424 13797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:46:52.889328 13802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:46:52.889351 13805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:46:52.889336 13803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:46:52.889557 13797 server_base.cc:1034] running on GCE node
I20260925 04:46:52.890213 13797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:46:52.891414 13797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:46:52.892675 13797 hybrid_clock.cc:648] HybridClock initialized: now 1790311612892661 us; error 39 us; skew 500 ppm
Sep 25 04:46:52 dist-test-slave-kghz krb5kdc[13709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311612, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:46:52.895730 13797 init.cc:377] Logged in from keytab as kudu/127.0.98.125@KRBTEST.COM (short username kudu)
I20260925 04:46:52.896729 13797 webserver.cc:460] Webserver started at http://127.0.98.125:37241/ using document root <none> and password file <none>
I20260925 04:46:52.897423 13797 fs_manager.cc:359] Metadata directory not provided
I20260925 04:46:52.897518 13797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:46:52.897735 13797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:46:52.900033 13797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311449873301-393-0/minicluster-data/master-1/data/instance:
uuid: "30d663a0d5064a98af9e6494dfa3b51c"
format_stamp: "Formatted at 2026-09-25 04:46:52 on dist-test-slave-kghz"
server_key: "8b1fde46c5e1bd746e0d7fc7872aecd2"
server_key_iv: "3f7530cad46cd2929d43f4c705cff10b"
server_key_version: "encryptionkey@0"
I20260925 04:46:52.900629 13797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311449873301-393-0/minicluster-data/master-1/wal/instance:
uuid: "30d663a0d5064a98af9e6494dfa3b51c"
format_stamp: "Formatted at 2026-09-25 04:46:52 on dist-test-slave-kghz"
server_key: "8b1fde46c5e1bd746e0d7fc7872aecd2"
server_key_iv: "3f7530cad46cd2929d43f4c705cff10b"
server_key_version: "encryptionkey@0"
I20260925 04:46:52.904448 13797 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 04:46:52.907008 13812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:46:52.908301 13797 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 04:46:52.908504 13797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311449873301-393-0/minicluster-data/master-1/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311449873301-393-0/minicluster-data/master-1/wal
uuid: "30d663a0d5064a98af9e6494dfa3b51c"
format_stamp: "Formatted at 2026-09-25 04:46:52 on dist-test-slave-kghz"
server_key: "8b1fde46c5e1bd746e0d7fc7872aecd2"
server_key_iv: "3f7530cad46cd2929d43f4c705cff10b"
server_key_version: "encryptionkey@0"
I20260925 04:46:52.908623 13797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311449873301-393-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311449873301-393-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311449873301-393-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:46:52.927688 13797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:46:52.931079 13797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:46:52.931298 13797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:46:52.938515 13797 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.125:46289
I20260925 04:46:52.938540 13864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.125:46289 every 8 connection(s)
I20260925 04:46:52.939527 13797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311449873301-393-0/minicluster-data/master-1/data/info.pb
I20260925 04:46:52.940303 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 13797
I20260925 04:46:52.940404 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311449873301-393-0/minicluster-data/master-1/wal/instance
I20260925 04:46:52.940681 393 external_mini_cluster.cc:1428] Setting key a135f46cefcb975e442755edad00c6f8
I20260925 04:46:52.942662 13865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:46:52.948758 13865 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33029 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.124@KRBTEST.COM" created.
Sep 25 04:46:52 dist-test-slave-kghz krb5kdc[13709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311612, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.125@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:46:52.964622 13794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:52.951402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50814 (local address 127.0.98.126:33029)
0925 04:46:52.951573 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 04:46:52.951577 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:46:52.952599 (+ 1022us) server_negotiation.cc:407] Connection header received
0925 04:46:52.953618 (+ 1019us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:46:52.953621 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:46:52.953686 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:46:52.953796 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:46:52.955507 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:52.956384 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:52.957721 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:52.958183 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:52.958346 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:52.961000 (+ 2654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:46:52.961004 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:46:52.961006 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:46:52.961033 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:46:52.962521 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:46:52.963084 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:46:52.963090 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:46:52.963094 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:46:52.963199 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:46:52.963528 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:46:52.963530 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:46:52.963531 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:46:52.963748 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:46:52.963825 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 04:46:52.964390 (+ 565us) server_negotiation.cc:299] Negotiation successful
0925 04:46:52.964499 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 04:46:52.965509 13868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:52.951859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:33029 (local address 127.0.0.1:50814)
0925 04:46:52.952446 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0925 04:46:52.952492 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 04:46:52.953394 (+ 902us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:46:52.954024 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:46:52.954037 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:46:52.954586 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:46:52.955336 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:52.955358 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:52.956546 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:52.956550 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:52.957579 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:52.957590 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:52.958515 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:52.958519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:52.958670 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:52.959299 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:46:52.959316 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:46:52.960812 (+ 1496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:46:52.962644 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:46:52.962654 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:46:52.962673 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:46:52.962978 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:46:52.963283 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:46:52.963285 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:46:52.963287 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:46:52.963417 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:46:52.963836 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:46:52.963841 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:46:52.964096 (+ 255us) client_negotiation.cc:769] Sending connection context
0925 04:46:52.964301 (+ 205us) client_negotiation.cc:240] Negotiation successful
0925 04:46:52.964638 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":150,"threads_started":1}
I20260925 04:46:52.967052 13865 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.125" port: 46289 } has no permanent_uuid. Determining permanent_uuid...
I20260925 04:46:52.967604 13790 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.125" port: 46289 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311449873301-393-0/krb5kdc/kudu_127.0.98.124.keytab.
Entry for principal kudu/127.0.98.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311449873301-393-0/krb5kdc/kudu_127.0.98.124.keytab.
I20260925 04:46:52.970472 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311449873301-393-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311449873301-393-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311449873301-393-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311449873301-393-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.124:43861
--webserver_interface=127.0.98.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311449873301-393-0/krb5kdc/kudu_127.0.98.124.keytab
--principal=kudu/127.0.98.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:45537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:33029,127.0.98.125:46289,127.0.98.124:43861
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 04:46:52 dist-test-slave-kghz krb5kdc[13709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311612, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for kudu/127.0.98.125@KRBTEST.COM
Sep 25 04:46:52 dist-test-slave-kghz krb5kdc[13709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311612, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.125@KRBTEST.COM for kudu/127.0.98.125@KRBTEST.COM
I20260925 04:46:52.980137 13793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:52.968001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:46289 (local address 127.0.0.1:38008)
0925 04:46:52.968156 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 04:46:52.968175 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 04:46:52.968251 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:46:52.968649 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:46:52.968652 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:46:52.968828 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:46:52.969395 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:52.969406 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:52.970965 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:52.970972 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:52.971910 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:52.971924 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:52.972752 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:52.972758 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:52.972884 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:52.973441 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:46:52.973461 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:46:52.975094 (+ 1633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:46:52.978224 (+ 3130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:46:52.978229 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:46:52.978231 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:46:52.978475 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:46:52.978817 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:46:52.978820 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:46:52.978822 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:46:52.978913 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:46:52.979615 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:46:52.979619 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:46:52.979728 (+ 109us) client_negotiation.cc:769] Sending connection context
0925 04:46:52.979797 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 04:46:52.979963 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260925 04:46:52.980314 13868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:52.967640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:46289 (local address 127.0.0.1:37994)
0925 04:46:52.967840 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0925 04:46:52.967852 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 04:46:52.967934 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:46:52.969564 (+ 1630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:46:52.969569 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:46:52.969725 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:46:52.969853 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:52.969863 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:52.972056 (+ 2193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:52.972059 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:52.972859 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:52.972866 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:52.973511 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:52.973513 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:52.973573 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:52.974020 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:46:52.974031 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:46:52.975655 (+ 1624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:46:52.978213 (+ 2558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:46:52.978217 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:46:52.978222 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:46:52.978554 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:46:52.978827 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:46:52.978833 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:46:52.978839 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:46:52.978956 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:46:52.979610 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:46:52.979615 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:46:52.979738 (+ 123us) client_negotiation.cc:769] Sending connection context
0925 04:46:52.979870 (+ 132us) client_negotiation.cc:240] Negotiation successful
0925 04:46:52.980042 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":20}
I20260925 04:46:52.980329 13870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:52.967737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37994 (local address 127.0.98.125:46289)
0925 04:46:52.968202 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0925 04:46:52.968211 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 04:46:52.968253 (+ 42us) server_negotiation.cc:407] Connection header received
0925 04:46:52.968299 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:46:52.968304 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:46:52.968512 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:46:52.968654 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:46:52.969994 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:52.970846 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:52.972974 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:52.973311 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:52.973418 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:52.975813 (+ 2395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:46:52.975816 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:46:52.975818 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:46:52.975837 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:46:52.978102 (+ 2265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:46:52.978659 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:46:52.978663 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:46:52.978664 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:46:52.978729 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:46:52.979060 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:46:52.979066 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:46:52.979071 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:46:52.979489 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:46:52.979629 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 04:46:52.979992 (+ 363us) server_negotiation.cc:299] Negotiation successful
0925 04:46:52.980134 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":176,"server-negotiator.queue_time_us":374,"spinlock_wait_cycles":14592,"thread_start_us":150,"threads_started":1}
I20260925 04:46:52.981175 13871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:52.968074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38008 (local address 127.0.98.125:46289)
0925 04:46:52.968344 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0925 04:46:52.968351 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:46:52.968361 (+ 10us) server_negotiation.cc:407] Connection header received
0925 04:46:52.968385 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:46:52.968387 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:46:52.968512 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:46:52.968654 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:46:52.969533 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:52.970846 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:52.972025 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:52.972614 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:52.972719 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:52.975260 (+ 2541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:46:52.975272 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:46:52.975292 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:46:52.975323 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:46:52.978085 (+ 2762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:46:52.978595 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:46:52.978602 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:46:52.978604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:46:52.978700 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:46:52.979006 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:46:52.979008 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:46:52.979009 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:46:52.979489 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:46:52.980850 (+ 1361us) server_negotiation.cc:1035] Waiting for connection context
0925 04:46:52.980918 (+ 68us) server_negotiation.cc:299] Negotiation successful
0925 04:46:52.981021 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":151,"server-negotiator.queue_time_us":218,"spinlock_wait_cycles":16640,"thread_start_us":123,"threads_started":1}
I20260925 04:46:52.982412 13790 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.124" port: 43861 } has no permanent_uuid. Determining permanent_uuid...
I20260925 04:46:52.982546 13865 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.124" port: 43861 } has no permanent_uuid. Determining permanent_uuid...
I20260925 04:46:52.983201 13793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:52.982943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:43861 (local address 127.0.0.1:36516)
0925 04:46:52.983076 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 04:46:52.983134 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:43861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260925 04:46:52.983508 13868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:52.982939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:43861 (local address 127.0.0.1:36526)
0925 04:46:52.983371 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0925 04:46:52.983444 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:43861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367}
W20260925 04:46:52.983734 13816 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.124:43861: connect: Connection refused (error 111)
I20260925 04:46:52.984565 13868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:52.984356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:43861 (local address 127.0.0.1:36528)
0925 04:46:52.984460 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0925 04:46:52.984510 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:43861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 04:46:52.984794 13793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:52.984534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:43861 (local address 127.0.0.1:36534)
0925 04:46:52.984678 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 04:46:52.984728 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:43861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260925 04:46:52.984812 13865 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.124:43861: Network error: Client connection negotiation failed: client connection to 127.0.98.124:43861: connect: Connection refused (error 111)
W20260925 04:46:52.984987 13790 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.124:43861: Network error: Client connection negotiation failed: client connection to 127.0.98.124:43861: connect: Connection refused (error 111)
I20260925 04:46:53.016150 13790 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.124" port: 43861 } attempt: 1
I20260925 04:46:53.017026 13793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:53.016721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:43861 (local address 127.0.0.1:36544)
0925 04:46:53.016878 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 04:46:53.016942 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:43861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260925 04:46:53.017762 13793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:53.017558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:43861 (local address 127.0.0.1:36550)
0925 04:46:53.017660 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0925 04:46:53.017713 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:43861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260925 04:46:53.017886 13790 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.124:43861: Network error: Client connection negotiation failed: client connection to 127.0.98.124:43861: connect: Connection refused (error 111)
I20260925 04:46:53.034103 13865 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.124" port: 43861 } attempt: 1
I20260925 04:46:53.035097 13868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:53.034714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:43861 (local address 127.0.0.1:36552)
0925 04:46:53.034897 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 04:46:53.034997 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:43861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260925 04:46:53.035938 13868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:53.035683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:43861 (local address 127.0.0.1:36560)
0925 04:46:53.035804 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 04:46:53.035859 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:43861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 04:46:53.036106 13865 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.124:43861: Network error: Client connection negotiation failed: client connection to 127.0.98.124:43861: connect: Connection refused (error 111)
W20260925 04:46:53.087282 13872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:46:53.087572 13872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:46:53.087615 13872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:46:53.091830 13872 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:46:53.091889 13872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:46:53.091910 13872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:46:53.091929 13872 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:46:53.091979 13872 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:46:53.093037 13790 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.124" port: 43861 } attempt: 2
I20260925 04:46:53.093827 13793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:53.093537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:43861 (local address 127.0.0.1:36572)
0925 04:46:53.093685 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 04:46:53.093752 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:43861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260925 04:46:53.094620 13793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:53.094379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:43861 (local address 127.0.0.1:36580)
0925 04:46:53.094523 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 04:46:53.094567 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:43861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260925 04:46:53.094739 13790 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.124:43861: Network error: Client connection negotiation failed: client connection to 127.0.98.124:43861: connect: Connection refused (error 111)
I20260925 04:46:53.097710 13872 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:45537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311449873301-393-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311449873301-393-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:33029,127.0.98.125:46289,127.0.98.124:43861
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311449873301-393-0/krb5kdc/kudu_127.0.98.124.keytab
--principal=kudu/127.0.98.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.124:43861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311449873301-393-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13872
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311449873301-393-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:46:53.098940 13872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:46:53.099994 13872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 04:46:53.104317 13865 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.124" port: 43861 } attempt: 2
I20260925 04:46:53.105217 13868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:53.104858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:43861 (local address 127.0.0.1:36582)
0925 04:46:53.105002 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 04:46:53.105095 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:43861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260925 04:46:53.105906 13868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:53.105691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:43861 (local address 127.0.0.1:36584)
0925 04:46:53.105796 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0925 04:46:53.105844 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:43861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260925 04:46:53.106047 13865 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.124:43861: Network error: Client connection negotiation failed: client connection to 127.0.98.124:43861: connect: Connection refused (error 111)
W20260925 04:46:53.108737 13879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:46:53.108805 13881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:46:53.108918 13878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:46:53.109176 13872 server_base.cc:1034] running on GCE node
I20260925 04:46:53.109721 13872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:46:53.110951 13872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:46:53.112121 13872 hybrid_clock.cc:648] HybridClock initialized: now 1790311613112103 us; error 27 us; skew 500 ppm
Sep 25 04:46:53 dist-test-slave-kghz krb5kdc[13709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311613, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:46:53.115224 13872 init.cc:377] Logged in from keytab as kudu/127.0.98.124@KRBTEST.COM (short username kudu)
I20260925 04:46:53.116205 13872 webserver.cc:460] Webserver started at http://127.0.98.124:43459/ using document root <none> and password file <none>
I20260925 04:46:53.116771 13872 fs_manager.cc:359] Metadata directory not provided
I20260925 04:46:53.116832 13872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:46:53.117060 13872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:46:53.119237 13872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311449873301-393-0/minicluster-data/master-2/data/instance:
uuid: "8c62150b7816495ca295a6b9700f8756"
format_stamp: "Formatted at 2026-09-25 04:46:53 on dist-test-slave-kghz"
server_key: "bee57a2c988cf893c98c885dd3265dbe"
server_key_iv: "d341e4c86e90781811c3ba9dd8d9db63"
server_key_version: "encryptionkey@0"
I20260925 04:46:53.119813 13872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311449873301-393-0/minicluster-data/master-2/wal/instance:
uuid: "8c62150b7816495ca295a6b9700f8756"
format_stamp: "Formatted at 2026-09-25 04:46:53 on dist-test-slave-kghz"
server_key: "bee57a2c988cf893c98c885dd3265dbe"
server_key_iv: "d341e4c86e90781811c3ba9dd8d9db63"
server_key_version: "encryptionkey@0"
I20260925 04:46:53.123602 13872 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 04:46:53.126055 13888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:46:53.127319 13872 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 04:46:53.127465 13872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311449873301-393-0/minicluster-data/master-2/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311449873301-393-0/minicluster-data/master-2/wal
uuid: "8c62150b7816495ca295a6b9700f8756"
format_stamp: "Formatted at 2026-09-25 04:46:53 on dist-test-slave-kghz"
server_key: "bee57a2c988cf893c98c885dd3265dbe"
server_key_iv: "d341e4c86e90781811c3ba9dd8d9db63"
server_key_version: "encryptionkey@0"
I20260925 04:46:53.127597 13872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311449873301-393-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311449873301-393-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311449873301-393-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:46:53.146054 13872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:46:53.149453 13872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:46:53.149662 13872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:46:53.156920 13872 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.124:43861
I20260925 04:46:53.156960 13940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.124:43861 every 8 connection(s)
I20260925 04:46:53.157896 13872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311449873301-393-0/minicluster-data/master-2/data/info.pb
I20260925 04:46:53.160724 13941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:46:53.164384 13941 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33029 } has no permanent_uuid. Determining permanent_uuid...
I20260925 04:46:53.168087 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 13872
I20260925 04:46:53.168182 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790311449873301-393-0/minicluster-data/master-2/wal/instance
I20260925 04:46:53.168493 393 external_mini_cluster.cc:1428] Setting key 94cf5006b2a6d2b9e3a6a277f90c7794
Sep 25 04:46:53 dist-test-slave-kghz krb5kdc[13709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311613, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.124@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
Sep 25 04:46:53 dist-test-slave-kghz krb5kdc[13709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311612, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.124@KRBTEST.COM
I20260925 04:46:53.179689 13794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:53.166356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50822 (local address 127.0.98.126:33029)
0925 04:46:53.166510 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 04:46:53.166514 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:46:53.167268 (+ 754us) server_negotiation.cc:407] Connection header received
0925 04:46:53.168279 (+ 1011us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:46:53.168282 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:46:53.168346 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:46:53.168456 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:46:53.170053 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:53.170929 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:53.172417 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:53.172788 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:53.172905 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:53.175664 (+ 2759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:46:53.175668 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:46:53.175673 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:46:53.175704 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:46:53.177328 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:46:53.177909 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:46:53.177914 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:46:53.177916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:46:53.178001 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:46:53.178413 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:46:53.178416 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:46:53.178418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:46:53.178664 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:46:53.178776 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 04:46:53.179451 (+ 675us) server_negotiation.cc:299] Negotiation successful
0925 04:46:53.179562 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 04:46:53.180729 13943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:53.166643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:33029 (local address 127.0.0.1:50822)
0925 04:46:53.167110 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0925 04:46:53.167151 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 04:46:53.168034 (+ 883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:46:53.168660 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:46:53.168668 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:46:53.169162 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:46:53.169852 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:53.169868 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:53.171056 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:53.171062 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:53.172249 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:53.172259 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:53.172976 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:53.172980 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:53.173143 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:53.173809 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:46:53.173830 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:46:53.175517 (+ 1687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:46:53.177513 (+ 1996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:46:53.177522 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:46:53.177533 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:46:53.177798 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:46:53.178110 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:46:53.178113 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:46:53.178115 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:46:53.178302 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:46:53.178808 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:46:53.178813 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:46:53.179138 (+ 325us) client_negotiation.cc:769] Sending connection context
0925 04:46:53.179333 (+ 195us) client_negotiation.cc:240] Negotiation successful
0925 04:46:53.179709 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":3,"thread_start_us":111,"threads_started":1}
I20260925 04:46:53.180742 13945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:53.169179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36596 (local address 127.0.98.124:43861)
0925 04:46:53.169555 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0925 04:46:53.169566 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 04:46:53.169590 (+ 24us) server_negotiation.cc:407] Connection header received
0925 04:46:53.169705 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:46:53.169713 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:46:53.169902 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:46:53.170064 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:46:53.170971 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:53.172304 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:53.173345 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:53.173882 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:53.174003 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:53.176729 (+ 2726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:46:53.176739 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:46:53.176755 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:46:53.176782 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:46:53.178506 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:46:53.178991 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:46:53.178996 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:46:53.178998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:46:53.179111 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:46:53.179395 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:46:53.179397 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:46:53.179399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:46:53.179661 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:46:53.179845 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0925 04:46:53.180121 (+ 276us) server_negotiation.cc:299] Negotiation successful
0925 04:46:53.180253 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":143,"threads_started":1}
I20260925 04:46:53.182179 13941 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.125" port: 46289 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 04:46:53 dist-test-slave-kghz krb5kdc[13709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311613, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.124@KRBTEST.COM for kudu/127.0.98.125@KRBTEST.COM
Sep 25 04:46:53 dist-test-slave-kghz krb5kdc[13709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311612, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:46:53.191656 13943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:53.182776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:46289 (local address 127.0.0.1:38020)
0925 04:46:53.182890 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0925 04:46:53.182904 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:46:53.182976 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:46:53.183396 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:46:53.183401 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:46:53.183599 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:46:53.183725 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:53.183734 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:53.184703 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:53.184706 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:53.185438 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:53.185447 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:53.185929 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:53.185931 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:53.186004 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:53.186575 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:46:53.186594 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:46:53.187894 (+ 1300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:46:53.190071 (+ 2177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:46:53.190073 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:46:53.190075 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:46:53.190306 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:46:53.190614 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:46:53.190617 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:46:53.190619 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:46:53.190709 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:46:53.191131 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:46:53.191136 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:46:53.191239 (+ 103us) client_negotiation.cc:769] Sending connection context
0925 04:46:53.191346 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 04:46:53.191494 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260925 04:46:53.191772 13871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:53.182895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38020 (local address 127.0.98.125:46289)
0925 04:46:53.183040 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 04:46:53.183046 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:46:53.183086 (+ 40us) server_negotiation.cc:407] Connection header received
0925 04:46:53.183132 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:46:53.183138 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:46:53.183189 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:46:53.183271 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:46:53.183857 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:53.184592 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:53.185559 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:53.185796 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:53.185924 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:53.188049 (+ 2125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:46:53.188052 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:46:53.188054 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:46:53.188077 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:46:53.189964 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:46:53.190412 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:46:53.190415 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:46:53.190417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:46:53.190507 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:46:53.190799 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:46:53.190801 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:46:53.190802 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:46:53.191034 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:46:53.191134 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 04:46:53.191451 (+ 317us) server_negotiation.cc:299] Negotiation successful
0925 04:46:53.191578 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 04:46:53.192627 13941 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.124" port: 43861 } has no permanent_uuid. Determining permanent_uuid...
I20260925 04:46:53.192682 13794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:53.183259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50836 (local address 127.0.98.126:33029)
0925 04:46:53.183362 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0925 04:46:53.183365 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:46:53.183380 (+ 15us) server_negotiation.cc:407] Connection header received
0925 04:46:53.183472 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:46:53.183474 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:46:53.183510 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:46:53.183574 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:46:53.184234 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:53.184895 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:53.185860 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:53.186149 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:53.186277 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:53.188708 (+ 2431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:46:53.188713 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:46:53.188716 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:46:53.188743 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:46:53.190434 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:46:53.190917 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:46:53.190920 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:46:53.190922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:46:53.190997 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:46:53.191286 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:46:53.191289 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:46:53.191290 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:46:53.191549 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:46:53.192331 (+ 782us) server_negotiation.cc:1035] Waiting for connection context
0925 04:46:53.192413 (+ 82us) server_negotiation.cc:299] Negotiation successful
0925 04:46:53.192527 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 04:46:53.193874 13790 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.124" port: 43861 } attempt: 3
I20260925 04:46:53.197198 13865 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.124" port: 43861 } attempt: 3
Sep 25 04:46:53 dist-test-slave-kghz krb5kdc[13709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311613, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.124@KRBTEST.COM for kudu/127.0.98.124@KRBTEST.COM
Sep 25 04:46:53 dist-test-slave-kghz krb5kdc[13709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311612, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.125@KRBTEST.COM
Sep 25 04:46:53 dist-test-slave-kghz krb5kdc[13709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311612, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for kudu/127.0.98.124@KRBTEST.COM
I20260925 04:46:53.202278 13945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:53.193198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36602 (local address 127.0.98.124:43861)
0925 04:46:53.193352 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 04:46:53.193356 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:46:53.193368 (+ 12us) server_negotiation.cc:407] Connection header received
0925 04:46:53.193429 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:46:53.193432 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:46:53.193474 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:46:53.193553 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:46:53.193971 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:53.194736 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:53.195692 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:53.195950 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:53.196059 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:53.198606 (+ 2547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:46:53.198610 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:46:53.198612 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:46:53.198633 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:46:53.200537 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:46:53.200948 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:46:53.200954 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:46:53.200958 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:46:53.201034 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:46:53.201281 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:46:53.201285 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:46:53.201288 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:46:53.201495 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:46:53.201598 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 04:46:53.201925 (+ 327us) server_negotiation.cc:299] Negotiation successful
0925 04:46:53.202046 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260925 04:46:53.202762 13943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:53.193152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:43861 (local address 127.0.0.1:36602)
0925 04:46:53.193251 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0925 04:46:53.193264 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:46:53.193352 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:46:53.193574 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:46:53.193576 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:46:53.193756 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:46:53.193855 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:53.193862 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:53.194847 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:53.194852 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:53.195581 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:53.195590 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:53.196055 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:53.196058 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:53.196131 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:53.196641 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:46:53.196655 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:46:53.198456 (+ 1801us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:46:53.200642 (+ 2186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:46:53.200644 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:46:53.200646 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:46:53.200856 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:46:53.201119 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:46:53.201122 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:46:53.201124 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:46:53.201196 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:46:53.201618 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:46:53.201624 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:46:53.201728 (+ 104us) client_negotiation.cc:769] Sending connection context
0925 04:46:53.201848 (+ 120us) client_negotiation.cc:240] Negotiation successful
0925 04:46:53.202000 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
Sep 25 04:46:53 dist-test-slave-kghz krb5kdc[13709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311612, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.125@KRBTEST.COM for kudu/127.0.98.124@KRBTEST.COM
I20260925 04:46:53.204953 13871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:53.193855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38022 (local address 127.0.98.125:46289)
0925 04:46:53.193967 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0925 04:46:53.193970 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:46:53.194179 (+ 209us) server_negotiation.cc:407] Connection header received
0925 04:46:53.194384 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:46:53.194389 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:46:53.194446 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:46:53.194499 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:46:53.195366 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:53.196163 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:53.197281 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:53.197590 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:53.197684 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:53.201000 (+ 3316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:46:53.201004 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:46:53.201006 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:46:53.201030 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:46:53.202961 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:46:53.203467 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:46:53.203470 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:46:53.203472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:46:53.203569 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:46:53.203884 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:46:53.203889 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:46:53.203894 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:46:53.204099 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:46:53.204279 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0925 04:46:53.204658 (+ 379us) server_negotiation.cc:299] Negotiation successful
0925 04:46:53.204773 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 04:46:53.205282 13793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:53.194222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:43861 (local address 127.0.0.1:36618)
0925 04:46:53.194375 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 04:46:53.194399 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 04:46:53.194469 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:46:53.194904 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:46:53.194907 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:46:53.195074 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:46:53.195147 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:53.195152 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:53.196572 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:53.196578 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:53.197103 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:53.197112 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:53.198629 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:53.198633 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:53.198733 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:53.199191 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:46:53.199206 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:46:53.200758 (+ 1552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:46:53.203712 (+ 2954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:46:53.203717 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:46:53.203719 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:46:53.203975 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:46:53.204291 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:46:53.204294 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:46:53.204295 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:46:53.204381 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:46:53.204833 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:46:53.204836 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:46:53.204941 (+ 105us) client_negotiation.cc:769] Sending connection context
0925 04:46:53.205018 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 04:46:53.205142 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260925 04:46:53.205899 13946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:53.194405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36618 (local address 127.0.98.124:43861)
0925 04:46:53.194682 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0925 04:46:53.194689 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:46:53.194700 (+ 11us) server_negotiation.cc:407] Connection header received
0925 04:46:53.194728 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:46:53.194730 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:46:53.194772 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:46:53.194887 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:46:53.195258 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:53.196444 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:53.197278 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:53.198505 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:53.198992 (+ 487us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:53.200966 (+ 1974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:46:53.200970 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:46:53.200973 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:46:53.200996 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:46:53.203598 (+ 2602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:46:53.204095 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:46:53.204100 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:46:53.204106 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:46:53.204190 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:46:53.204491 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:46:53.204493 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:46:53.204495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:46:53.204716 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:46:53.205545 (+ 829us) server_negotiation.cc:1035] Waiting for connection context
0925 04:46:53.205615 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 04:46:53.205715 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":220,"thread_start_us":123,"threads_started":1}
I20260925 04:46:53.209141 13790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa6306245adf42af962e1a467e6724af: Bootstrap starting.
I20260925 04:46:53.210559 393 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:46:53.212123 13790 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa6306245adf42af962e1a467e6724af: Neither blocks nor log segments found. Creating new log.
I20260925 04:46:53.212355 13868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:53.197587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:43861 (local address 127.0.0.1:36624)
0925 04:46:53.197720 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 04:46:53.197733 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:46:53.197800 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:46:53.198454 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:46:53.198460 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:46:53.198638 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:46:53.198748 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:53.198758 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:53.200051 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:53.200055 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:53.200906 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:53.200918 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:53.202204 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:53.202207 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:53.202334 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:53.202977 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:46:53.202991 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:46:53.204508 (+ 1517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:46:53.210649 (+ 6141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:46:53.210652 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:46:53.210655 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:46:53.210911 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:46:53.211261 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:46:53.211265 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:46:53.211268 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:46:53.211355 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:46:53.211817 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:46:53.211820 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:46:53.211933 (+ 113us) client_negotiation.cc:769] Sending connection context
0925 04:46:53.212041 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 04:46:53.212205 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260925 04:46:53.213141 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:53.197804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36624 (local address 127.0.98.124:43861)
0925 04:46:53.198142 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0925 04:46:53.198184 (+ 42us) server_negotiation.cc:399] Waiting for connection header
0925 04:46:53.198201 (+ 17us) server_negotiation.cc:407] Connection header received
0925 04:46:53.198237 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:46:53.198242 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:46:53.198315 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:46:53.198876 (+ 561us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:46:53.199086 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:53.199924 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:53.201059 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:53.201811 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:53.204631 (+ 2820us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:53.204729 (+ 98us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:46:53.204735 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:46:53.204740 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:46:53.204757 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:46:53.210498 (+ 5741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:46:53.211058 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:46:53.211062 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:46:53.211064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:46:53.211156 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:46:53.211479 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:46:53.211481 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:46:53.211483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:46:53.211704 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:46:53.212787 (+ 1083us) server_negotiation.cc:1035] Waiting for connection context
0925 04:46:53.212866 (+ 79us) server_negotiation.cc:299] Negotiation successful
0925 04:46:53.212975 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":277,"thread_start_us":278,"threads_started":1}
I20260925 04:46:53.213236 13790 log.cc:824] T 00000000000000000000000000000000 P aa6306245adf42af962e1a467e6724af: Log is configured to *not* fsync() on all Append() calls
I20260925 04:46:53.213936 13941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c62150b7816495ca295a6b9700f8756: Bootstrap starting.
I20260925 04:46:53.215612 13790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa6306245adf42af962e1a467e6724af: No bootstrap required, opened a new log
I20260925 04:46:53.216101 13865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30d663a0d5064a98af9e6494dfa3b51c: Bootstrap starting.
I20260925 04:46:53.216627 13941 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c62150b7816495ca295a6b9700f8756: Neither blocks nor log segments found. Creating new log.
I20260925 04:46:53.217474 13941 log.cc:824] T 00000000000000000000000000000000 P 8c62150b7816495ca295a6b9700f8756: Log is configured to *not* fsync() on all Append() calls
I20260925 04:46:53.218899 13865 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30d663a0d5064a98af9e6494dfa3b51c: Neither blocks nor log segments found. Creating new log.
I20260925 04:46:53.219617 13941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c62150b7816495ca295a6b9700f8756: No bootstrap required, opened a new log
I20260925 04:46:53.219723 13865 log.cc:824] T 00000000000000000000000000000000 P 30d663a0d5064a98af9e6494dfa3b51c: Log is configured to *not* fsync() on all Append() calls
I20260925 04:46:53.219777 13790 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa6306245adf42af962e1a467e6724af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa6306245adf42af962e1a467e6724af" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33029 } } peers { permanent_uuid: "30d663a0d5064a98af9e6494dfa3b51c" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 46289 } } peers { permanent_uuid: "8c62150b7816495ca295a6b9700f8756" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 43861 } }
I20260925 04:46:53.220026 13790 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa6306245adf42af962e1a467e6724af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:46:53.220084 13790 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa6306245adf42af962e1a467e6724af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa6306245adf42af962e1a467e6724af, State: Initialized, Role: FOLLOWER
I20260925 04:46:53.220618 13790 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa6306245adf42af962e1a467e6724af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa6306245adf42af962e1a467e6724af" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33029 } } peers { permanent_uuid: "30d663a0d5064a98af9e6494dfa3b51c" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 46289 } } peers { permanent_uuid: "8c62150b7816495ca295a6b9700f8756" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 43861 } }
I20260925 04:46:53.221115 13958 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa6306245adf42af962e1a467e6724af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa6306245adf42af962e1a467e6724af" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33029 } } peers { permanent_uuid: "30d663a0d5064a98af9e6494dfa3b51c" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 46289 } } peers { permanent_uuid: "8c62150b7816495ca295a6b9700f8756" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 43861 } } }
I20260925 04:46:53.221257 13958 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa6306245adf42af962e1a467e6724af [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:46:53.221937 13790 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa6306245adf42af962e1a467e6724af [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:46:53.223948 13941 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c62150b7816495ca295a6b9700f8756 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa6306245adf42af962e1a467e6724af" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33029 } } peers { permanent_uuid: "30d663a0d5064a98af9e6494dfa3b51c" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 46289 } } peers { permanent_uuid: "8c62150b7816495ca295a6b9700f8756" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 43861 } }
I20260925 04:46:53.224205 13941 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c62150b7816495ca295a6b9700f8756 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:46:53.224258 13941 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c62150b7816495ca295a6b9700f8756 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c62150b7816495ca295a6b9700f8756, State: Initialized, Role: FOLLOWER
I20260925 04:46:53.224699 13941 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c62150b7816495ca295a6b9700f8756 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa6306245adf42af962e1a467e6724af" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33029 } } peers { permanent_uuid: "30d663a0d5064a98af9e6494dfa3b51c" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 46289 } } peers { permanent_uuid: "8c62150b7816495ca295a6b9700f8756" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 43861 } }
I20260925 04:46:53.225164 13970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c62150b7816495ca295a6b9700f8756 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa6306245adf42af962e1a467e6724af" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33029 } } peers { permanent_uuid: "30d663a0d5064a98af9e6494dfa3b51c" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 46289 } } peers { permanent_uuid: "8c62150b7816495ca295a6b9700f8756" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 43861 } } }
I20260925 04:46:53.225339 13970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c62150b7816495ca295a6b9700f8756 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:46:53.225973 13941 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c62150b7816495ca295a6b9700f8756 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:46:53.229840 13871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:53.213360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38028 (local address 127.0.98.125:46289)
0925 04:46:53.213476 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0925 04:46:53.213480 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:46:53.219486 (+ 6006us) server_negotiation.cc:407] Connection header received
0925 04:46:53.219714 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:46:53.219718 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:46:53.219767 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:46:53.219872 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:46:53.221101 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:53.221886 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:53.223196 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:53.223548 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:53.223659 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:53.225470 (+ 1811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:46:53.225474 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:46:53.225476 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:46:53.225504 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:46:53.227346 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:46:53.227854 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:46:53.227858 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:46:53.227860 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:46:53.227950 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:46:53.228263 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:46:53.228267 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:46:53.228269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:46:53.228475 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:46:53.228601 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0925 04:46:53.229580 (+ 979us) server_negotiation.cc:299] Negotiation successful
0925 04:46:53.229693 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 04:46:53.227430 13722 master_runner.cc:186] Error getting master registration for 127.0.98.125:46289: OK, instance_id { permanent_uuid: "30d663a0d5064a98af9e6494dfa3b51c" instance_seqno: 1790311612936328 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 04:46:53.232999 13722 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 04:46:53.233232 13865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30d663a0d5064a98af9e6494dfa3b51c: No bootstrap required, opened a new log
I20260925 04:46:53.238144 13865 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30d663a0d5064a98af9e6494dfa3b51c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa6306245adf42af962e1a467e6724af" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33029 } } peers { permanent_uuid: "30d663a0d5064a98af9e6494dfa3b51c" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 46289 } } peers { permanent_uuid: "8c62150b7816495ca295a6b9700f8756" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 43861 } }
I20260925 04:46:53.238408 13865 raft_consensus.cc:374] T 00000000000000000000000000000000 P 30d663a0d5064a98af9e6494dfa3b51c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:46:53.238467 13865 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30d663a0d5064a98af9e6494dfa3b51c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30d663a0d5064a98af9e6494dfa3b51c, State: Initialized, Role: FOLLOWER
I20260925 04:46:53.234943 13872 master_runner.cc:186] Error getting master registration for 127.0.98.125:46289: OK, instance_id { permanent_uuid: "30d663a0d5064a98af9e6494dfa3b51c" instance_seqno: 1790311612936328 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 04:46:53.238941 13872 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260925 04:46:53.231283 13981 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8c62150b7816495ca295a6b9700f8756: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 04:46:53.239006 13981 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 04:46:53.239002 13865 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30d663a0d5064a98af9e6494dfa3b51c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa6306245adf42af962e1a467e6724af" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33029 } } peers { permanent_uuid: "30d663a0d5064a98af9e6494dfa3b51c" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 46289 } } peers { permanent_uuid: "8c62150b7816495ca295a6b9700f8756" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 43861 } }
I20260925 04:46:53.240338 13865 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30d663a0d5064a98af9e6494dfa3b51c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:46:53.242342 13794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:53.213712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50846 (local address 127.0.98.126:33029)
0925 04:46:53.217467 (+ 3755us) server_negotiation.cc:206] Beginning negotiation
0925 04:46:53.217472 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:46:53.230441 (+ 12969us) server_negotiation.cc:407] Connection header received
0925 04:46:53.230487 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:46:53.230490 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:46:53.230535 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:46:53.230628 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:46:53.233915 (+ 3287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:53.234637 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:53.235724 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:53.236042 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:53.236112 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:53.238726 (+ 2614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:46:53.238730 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:46:53.238732 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:46:53.238758 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:46:53.240310 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:46:53.240876 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:46:53.240879 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:46:53.240881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:46:53.240981 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:46:53.241304 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:46:53.241307 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:46:53.241309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:46:53.241513 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:46:53.241638 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 04:46:53.242014 (+ 376us) server_negotiation.cc:299] Negotiation successful
0925 04:46:53.242135 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3675}
I20260925 04:46:53.243062 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:53.218837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36632 (local address 127.0.98.124:43861)
0925 04:46:53.218937 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0925 04:46:53.218944 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:46:53.229143 (+ 10199us) server_negotiation.cc:407] Connection header received
0925 04:46:53.229188 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:46:53.229191 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:46:53.229236 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:46:53.229311 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:46:53.231074 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:53.231805 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:53.233226 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:53.233563 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:53.233645 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:53.235597 (+ 1952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:46:53.235603 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:46:53.235606 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:46:53.235631 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:46:53.237502 (+ 1871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:46:53.240876 (+ 3374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:46:53.240879 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:46:53.240881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:46:53.240981 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:46:53.241304 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:46:53.241307 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:46:53.241309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:46:53.241513 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:46:53.241644 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 04:46:53.242802 (+ 1158us) server_negotiation.cc:299] Negotiation successful
0925 04:46:53.242919 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260925 04:46:53.245129 13969 catalog_manager.cc:1558] T 00000000000000000000000000000000 P aa6306245adf42af962e1a467e6724af: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 04:46:53.245208 13969 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 04:46:53.248111 13797 master_runner.cc:263] No leader master found from master 30d663a0d5064a98af9e6494dfa3b51c
I20260925 04:46:53.248172 13797 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 04:46:53.248322 13984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30d663a0d5064a98af9e6494dfa3b51c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa6306245adf42af962e1a467e6724af" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33029 } } peers { permanent_uuid: "30d663a0d5064a98af9e6494dfa3b51c" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 46289 } } peers { permanent_uuid: "8c62150b7816495ca295a6b9700f8756" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 43861 } } }
I20260925 04:46:53.248421 13984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30d663a0d5064a98af9e6494dfa3b51c [sys.catalog]: This master's current role is: FOLLOWER
W20260925 04:46:53.250202 13995 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 30d663a0d5064a98af9e6494dfa3b51c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 04:46:53.250298 13995 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 04:46:53.311457 13984 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30d663a0d5064a98af9e6494dfa3b51c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:46:53.311662 13984 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30d663a0d5064a98af9e6494dfa3b51c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa6306245adf42af962e1a467e6724af" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33029 } } peers { permanent_uuid: "30d663a0d5064a98af9e6494dfa3b51c" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 46289 } } peers { permanent_uuid: "8c62150b7816495ca295a6b9700f8756" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 43861 } }
I20260925 04:46:53.312764 13984 leader_election.cc:288] T 00000000000000000000000000000000 P 30d663a0d5064a98af9e6494dfa3b51c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aa6306245adf42af962e1a467e6724af (127.0.98.126:33029), 8c62150b7816495ca295a6b9700f8756 (127.0.98.124:43861)
I20260925 04:46:53.313702 13764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "30d663a0d5064a98af9e6494dfa3b51c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa6306245adf42af962e1a467e6724af" is_pre_election: true
I20260925 04:46:53.314008 13764 raft_consensus.cc:2454] T 00000000000000000000000000000000 P aa6306245adf42af962e1a467e6724af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30d663a0d5064a98af9e6494dfa3b51c in term 0.
I20260925 04:46:53.314450 13915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "30d663a0d5064a98af9e6494dfa3b51c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c62150b7816495ca295a6b9700f8756" is_pre_election: true
I20260925 04:46:53.314738 13915 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8c62150b7816495ca295a6b9700f8756 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30d663a0d5064a98af9e6494dfa3b51c in term 0.
I20260925 04:46:53.315255 13816 leader_election.cc:302] T 00000000000000000000000000000000 P 30d663a0d5064a98af9e6494dfa3b51c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30d663a0d5064a98af9e6494dfa3b51c, 8c62150b7816495ca295a6b9700f8756; no voters:
I20260925 04:46:53.315754 13984 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30d663a0d5064a98af9e6494dfa3b51c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:46:53.315841 13984 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30d663a0d5064a98af9e6494dfa3b51c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:46:53.315889 13984 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30d663a0d5064a98af9e6494dfa3b51c [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:46:53.317291 13984 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30d663a0d5064a98af9e6494dfa3b51c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa6306245adf42af962e1a467e6724af" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33029 } } peers { permanent_uuid: "30d663a0d5064a98af9e6494dfa3b51c" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 46289 } } peers { permanent_uuid: "8c62150b7816495ca295a6b9700f8756" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 43861 } }
I20260925 04:46:53.317659 13984 leader_election.cc:288] T 00000000000000000000000000000000 P 30d663a0d5064a98af9e6494dfa3b51c [CANDIDATE]: Term 1 election: Requested vote from peers aa6306245adf42af962e1a467e6724af (127.0.98.126:33029), 8c62150b7816495ca295a6b9700f8756 (127.0.98.124:43861)
I20260925 04:46:53.318312 13915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "30d663a0d5064a98af9e6494dfa3b51c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c62150b7816495ca295a6b9700f8756"
I20260925 04:46:53.318446 13915 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c62150b7816495ca295a6b9700f8756 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:46:53.319756 13915 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8c62150b7816495ca295a6b9700f8756 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30d663a0d5064a98af9e6494dfa3b51c in term 1.
I20260925 04:46:53.320089 13764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "30d663a0d5064a98af9e6494dfa3b51c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa6306245adf42af962e1a467e6724af"
I20260925 04:46:53.320238 13764 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa6306245adf42af962e1a467e6724af [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:46:53.321518 13764 raft_consensus.cc:2454] T 00000000000000000000000000000000 P aa6306245adf42af962e1a467e6724af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30d663a0d5064a98af9e6494dfa3b51c in term 1.
I20260925 04:46:53.321843 13816 leader_election.cc:302] T 00000000000000000000000000000000 P 30d663a0d5064a98af9e6494dfa3b51c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30d663a0d5064a98af9e6494dfa3b51c, aa6306245adf42af962e1a467e6724af; no voters:
I20260925 04:46:53.322233 13984 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30d663a0d5064a98af9e6494dfa3b51c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:46:53.322494 13984 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30d663a0d5064a98af9e6494dfa3b51c [term 1 LEADER]: Becoming Leader. State: Replica: 30d663a0d5064a98af9e6494dfa3b51c, State: Running, Role: LEADER
I20260925 04:46:53.322852 13984 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30d663a0d5064a98af9e6494dfa3b51c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa6306245adf42af962e1a467e6724af" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33029 } } peers { permanent_uuid: "30d663a0d5064a98af9e6494dfa3b51c" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 46289 } } peers { permanent_uuid: "8c62150b7816495ca295a6b9700f8756" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 43861 } }
I20260925 04:46:53.324818 13984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30d663a0d5064a98af9e6494dfa3b51c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30d663a0d5064a98af9e6494dfa3b51c. Latest consensus state: current_term: 1 leader_uuid: "30d663a0d5064a98af9e6494dfa3b51c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa6306245adf42af962e1a467e6724af" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33029 } } peers { permanent_uuid: "30d663a0d5064a98af9e6494dfa3b51c" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 46289 } } peers { permanent_uuid: "8c62150b7816495ca295a6b9700f8756" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 43861 } } }
I20260925 04:46:53.324934 13984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30d663a0d5064a98af9e6494dfa3b51c [sys.catalog]: This master's current role is: LEADER
I20260925 04:46:53.325321 13999 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:46:53.326381 13999 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:46:53.334226 13915 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8c62150b7816495ca295a6b9700f8756 [term 1 FOLLOWER]: Refusing update from remote peer 30d663a0d5064a98af9e6494dfa3b51c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:46:53.334932 13997 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 30d663a0d5064a98af9e6494dfa3b51c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c62150b7816495ca295a6b9700f8756" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 43861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:46:53.339097 13970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c62150b7816495ca295a6b9700f8756 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30d663a0d5064a98af9e6494dfa3b51c. Latest consensus state: current_term: 1 leader_uuid: "30d663a0d5064a98af9e6494dfa3b51c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa6306245adf42af962e1a467e6724af" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33029 } } peers { permanent_uuid: "30d663a0d5064a98af9e6494dfa3b51c" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 46289 } } peers { permanent_uuid: "8c62150b7816495ca295a6b9700f8756" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 43861 } } }
I20260925 04:46:53.339250 13970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c62150b7816495ca295a6b9700f8756 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:46:53.340266 13764 raft_consensus.cc:1261] T 00000000000000000000000000000000 P aa6306245adf42af962e1a467e6724af [term 1 FOLLOWER]: Refusing update from remote peer 30d663a0d5064a98af9e6494dfa3b51c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:46:53.340919 13997 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 30d663a0d5064a98af9e6494dfa3b51c [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa6306245adf42af962e1a467e6724af" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:46:53.345620 13958 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa6306245adf42af962e1a467e6724af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30d663a0d5064a98af9e6494dfa3b51c. Latest consensus state: current_term: 1 leader_uuid: "30d663a0d5064a98af9e6494dfa3b51c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa6306245adf42af962e1a467e6724af" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33029 } } peers { permanent_uuid: "30d663a0d5064a98af9e6494dfa3b51c" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 46289 } } peers { permanent_uuid: "8c62150b7816495ca295a6b9700f8756" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 43861 } } }
I20260925 04:46:53.345757 13958 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa6306245adf42af962e1a467e6724af [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:46:53.352399 13997 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30d663a0d5064a98af9e6494dfa3b51c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "30d663a0d5064a98af9e6494dfa3b51c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa6306245adf42af962e1a467e6724af" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33029 } } peers { permanent_uuid: "30d663a0d5064a98af9e6494dfa3b51c" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 46289 } } peers { permanent_uuid: "8c62150b7816495ca295a6b9700f8756" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 43861 } } }
I20260925 04:46:53.352535 13997 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30d663a0d5064a98af9e6494dfa3b51c [sys.catalog]: This master's current role is: LEADER
I20260925 04:46:53.353734 13958 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa6306245adf42af962e1a467e6724af [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "30d663a0d5064a98af9e6494dfa3b51c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa6306245adf42af962e1a467e6724af" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33029 } } peers { permanent_uuid: "30d663a0d5064a98af9e6494dfa3b51c" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 46289 } } peers { permanent_uuid: "8c62150b7816495ca295a6b9700f8756" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 43861 } } }
I20260925 04:46:53.353856 13958 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa6306245adf42af962e1a467e6724af [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:46:53.354233 14000 mvcc.cc:205] Tried to move back new op lower bound from 7333116368209133568 to 7333116368174010368. Current Snapshot: MvccSnapshot[applied={T|T < 7333116368209133568}]
I20260925 04:46:53.354429 14003 mvcc.cc:205] Tried to move back new op lower bound from 7333116368209133568 to 7333116368174010368. Current Snapshot: MvccSnapshot[applied={T|T < 7333116368209133568}]
I20260925 04:46:53.355345 13984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30d663a0d5064a98af9e6494dfa3b51c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "30d663a0d5064a98af9e6494dfa3b51c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa6306245adf42af962e1a467e6724af" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33029 } } peers { permanent_uuid: "30d663a0d5064a98af9e6494dfa3b51c" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 46289 } } peers { permanent_uuid: "8c62150b7816495ca295a6b9700f8756" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 43861 } } }
I20260925 04:46:53.355460 13984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30d663a0d5064a98af9e6494dfa3b51c [sys.catalog]: This master's current role is: LEADER
I20260925 04:46:53.356352 13970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c62150b7816495ca295a6b9700f8756 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "30d663a0d5064a98af9e6494dfa3b51c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa6306245adf42af962e1a467e6724af" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33029 } } peers { permanent_uuid: "30d663a0d5064a98af9e6494dfa3b51c" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 46289 } } peers { permanent_uuid: "8c62150b7816495ca295a6b9700f8756" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 43861 } } }
I20260925 04:46:53.356479 13970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c62150b7816495ca295a6b9700f8756 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:46:53.361639 13999 catalog_manager.cc:1347] Generated new cluster ID: ce267a4453964aba89dcb8ad1d75475e
I20260925 04:46:53.361723 13999 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:46:53.361860 14001 mvcc.cc:205] Tried to move back new op lower bound from 7333116368209133568 to 7333116368174010368. Current Snapshot: MvccSnapshot[applied={T|T < 7333116368209133568}]
I20260925 04:46:53.387218 13999 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:46:53.388458 13999 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:46:53.418789 13999 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 30d663a0d5064a98af9e6494dfa3b51c: Generated new TSK 0
I20260925 04:46:53.419744 13999 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:46:54.240445 13981 catalog_manager.cc:1259] Loaded cluster ID: ce267a4453964aba89dcb8ad1d75475e
I20260925 04:46:54.240528 13981 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8c62150b7816495ca295a6b9700f8756: loading cluster ID for follower catalog manager: success
I20260925 04:46:54.242441 13981 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8c62150b7816495ca295a6b9700f8756: acquiring CA information for follower catalog manager: success
I20260925 04:46:54.243551 13981 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8c62150b7816495ca295a6b9700f8756: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 04:46:54.246341 13969 catalog_manager.cc:1259] Loaded cluster ID: ce267a4453964aba89dcb8ad1d75475e
I20260925 04:46:54.246454 13969 catalog_manager.cc:1552] T 00000000000000000000000000000000 P aa6306245adf42af962e1a467e6724af: loading cluster ID for follower catalog manager: success
I20260925 04:46:54.248008 13969 catalog_manager.cc:1574] T 00000000000000000000000000000000 P aa6306245adf42af962e1a467e6724af: acquiring CA information for follower catalog manager: success
I20260925 04:46:54.249048 13969 catalog_manager.cc:1602] T 00000000000000000000000000000000 P aa6306245adf42af962e1a467e6724af: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 04:46:54.500909 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 13722
I20260925 04:46:54.507215 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 13797
I20260925 04:46:54.513193 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 13872
2026-09-25T04:46:54Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3142 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790311449873301-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:46:54 dist-test-slave-kghz krb5kdc[14014](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:46:54 dist-test-slave-kghz krb5kdc[14014](info): set up 2 sockets
Sep 25 04:46:54 dist-test-slave-kghz krb5kdc[14014](info): commencing operation
krb5kdc: starting...
W20260925 04:46:55.550523 393 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790311449873301-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790311449873301-393-0/krb5kdc/test-user.keytab.
Sep 25 04:46:55 dist-test-slave-kghz krb5kdc[14014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311615, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:46:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:46:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260925 04:46:55.674350 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790311449873301-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790311449873301-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790311449873301-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:40367
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:43939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:40367
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:46:55.785120 14027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:46:55.785388 14027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:46:55.785429 14027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:46:55.789426 14027 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:46:55.789494 14027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:46:55.789512 14027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:46:55.789529 14027 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:46:55.789583 14027 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:46:55.794926 14027 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:43939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790311449873301-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790311449873301-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:40367
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:40367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790311449873301-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:46:55.796027 14027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:46:55.797075 14027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 04:46:55.805850 14027 server_base.cc:1034] running on GCE node
W20260925 04:46:55.805930 14033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:46:55.805856 14032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:46:55.806067 14035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:46:55.806865 14027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:46:55.807982 14027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:46:55.809228 14027 hybrid_clock.cc:648] HybridClock initialized: now 1790311615809212 us; error 28 us; skew 500 ppm
Sep 25 04:46:55 dist-test-slave-kghz krb5kdc[14014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311615, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:46:55.812275 14027 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260925 04:46:55.813146 14027 webserver.cc:460] Webserver started at http://127.0.98.126:43707/ using document root <none> and password file <none>
I20260925 04:46:55.813697 14027 fs_manager.cc:359] Metadata directory not provided
I20260925 04:46:55.813756 14027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:46:55.813921 14027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:46:55.816009 14027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790311449873301-393-0/minicluster-data/master-0/data/instance:
uuid: "51a2a4c6d9504c1aabd1f8e9abfe87bb"
format_stamp: "Formatted at 2026-09-25 04:46:55 on dist-test-slave-kghz"
server_key: "323070ac91ac4c36d83c9ea3ff5940b4"
server_key_iv: "9310faaca19f70a08bf94470027e2f8f"
server_key_version: "encryptionkey@0"
I20260925 04:46:55.816546 14027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790311449873301-393-0/minicluster-data/master-0/wal/instance:
uuid: "51a2a4c6d9504c1aabd1f8e9abfe87bb"
format_stamp: "Formatted at 2026-09-25 04:46:55 on dist-test-slave-kghz"
server_key: "323070ac91ac4c36d83c9ea3ff5940b4"
server_key_iv: "9310faaca19f70a08bf94470027e2f8f"
server_key_version: "encryptionkey@0"
I20260925 04:46:55.820437 14027 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260925 04:46:55.822928 14042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:46:55.824215 14027 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 04:46:55.824319 14027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790311449873301-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790311449873301-393-0/minicluster-data/master-0/wal
uuid: "51a2a4c6d9504c1aabd1f8e9abfe87bb"
format_stamp: "Formatted at 2026-09-25 04:46:55 on dist-test-slave-kghz"
server_key: "323070ac91ac4c36d83c9ea3ff5940b4"
server_key_iv: "9310faaca19f70a08bf94470027e2f8f"
server_key_version: "encryptionkey@0"
I20260925 04:46:55.824442 14027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790311449873301-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790311449873301-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790311449873301-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:46:55.836841 14027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:46:55.840167 14027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:46:55.840345 14027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:46:55.847290 14027 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:40367
I20260925 04:46:55.847379 14094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:40367 every 8 connection(s)
I20260925 04:46:55.848285 14027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
I20260925 04:46:55.850522 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 14027
I20260925 04:46:55.850621 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790311449873301-393-0/minicluster-data/master-0/wal/instance
I20260925 04:46:55.850927 393 external_mini_cluster.cc:1428] Setting key 181a5a86bb86661cf216b489d5736a9e
I20260925 04:46:55.851227 14095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:46:55.858417 14095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51a2a4c6d9504c1aabd1f8e9abfe87bb: Bootstrap starting.
Sep 25 04:46:55 dist-test-slave-kghz krb5kdc[14014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311615, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:46:55.860649 14095 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 51a2a4c6d9504c1aabd1f8e9abfe87bb: Neither blocks nor log segments found. Creating new log.
I20260925 04:46:55.861560 14095 log.cc:824] T 00000000000000000000000000000000 P 51a2a4c6d9504c1aabd1f8e9abfe87bb: Log is configured to *not* fsync() on all Append() calls
I20260925 04:46:55.863896 14095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51a2a4c6d9504c1aabd1f8e9abfe87bb: No bootstrap required, opened a new log
I20260925 04:46:55.865695 14098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:55.852295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57374 (local address 127.0.98.126:40367)
0925 04:46:55.852749 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0925 04:46:55.852777 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0925 04:46:55.852826 (+ 49us) server_negotiation.cc:407] Connection header received
0925 04:46:55.853607 (+ 781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:46:55.853629 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:46:55.854067 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:46:55.854459 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:46:55.855142 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:55.856607 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:55.857498 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:55.858157 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:55.858335 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:55.860958 (+ 2623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:46:55.860965 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:46:55.860981 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:46:55.861009 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:46:55.862873 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:46:55.863380 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:46:55.863384 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:46:55.863388 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:46:55.863495 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:46:55.863776 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:46:55.863778 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:46:55.863779 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:46:55.864232 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:46:55.864399 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 04:46:55.864701 (+ 302us) server_negotiation.cc:299] Negotiation successful
0925 04:46:55.864993 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":113,"threads_started":1}
I20260925 04:46:55.867697 14095 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51a2a4c6d9504c1aabd1f8e9abfe87bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51a2a4c6d9504c1aabd1f8e9abfe87bb" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 40367 } }
I20260925 04:46:55.867976 14095 raft_consensus.cc:374] T 00000000000000000000000000000000 P 51a2a4c6d9504c1aabd1f8e9abfe87bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:46:55.868053 14095 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51a2a4c6d9504c1aabd1f8e9abfe87bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51a2a4c6d9504c1aabd1f8e9abfe87bb, State: Initialized, Role: FOLLOWER
I20260925 04:46:55.868639 14095 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51a2a4c6d9504c1aabd1f8e9abfe87bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51a2a4c6d9504c1aabd1f8e9abfe87bb" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 40367 } }
I20260925 04:46:55.868836 14095 raft_consensus.cc:388] T 00000000000000000000000000000000 P 51a2a4c6d9504c1aabd1f8e9abfe87bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:46:55.868923 14095 raft_consensus.cc:482] T 00000000000000000000000000000000 P 51a2a4c6d9504c1aabd1f8e9abfe87bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:46:55.869025 14095 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51a2a4c6d9504c1aabd1f8e9abfe87bb [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:46:55.870505 14095 raft_consensus.cc:504] T 00000000000000000000000000000000 P 51a2a4c6d9504c1aabd1f8e9abfe87bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51a2a4c6d9504c1aabd1f8e9abfe87bb" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 40367 } }
I20260925 04:46:55.870887 14095 leader_election.cc:302] T 00000000000000000000000000000000 P 51a2a4c6d9504c1aabd1f8e9abfe87bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51a2a4c6d9504c1aabd1f8e9abfe87bb; no voters:
I20260925 04:46:55.871268 14095 leader_election.cc:288] T 00000000000000000000000000000000 P 51a2a4c6d9504c1aabd1f8e9abfe87bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:46:55.871376 14100 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 51a2a4c6d9504c1aabd1f8e9abfe87bb [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:46:55.871620 14100 raft_consensus.cc:686] T 00000000000000000000000000000000 P 51a2a4c6d9504c1aabd1f8e9abfe87bb [term 1 LEADER]: Becoming Leader. State: Replica: 51a2a4c6d9504c1aabd1f8e9abfe87bb, State: Running, Role: LEADER
I20260925 04:46:55.871997 14100 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51a2a4c6d9504c1aabd1f8e9abfe87bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51a2a4c6d9504c1aabd1f8e9abfe87bb" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 40367 } }
I20260925 04:46:55.872596 14095 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51a2a4c6d9504c1aabd1f8e9abfe87bb [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:46:55.872957 393 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:46:55.875020 14101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51a2a4c6d9504c1aabd1f8e9abfe87bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "51a2a4c6d9504c1aabd1f8e9abfe87bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51a2a4c6d9504c1aabd1f8e9abfe87bb" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 40367 } } }
I20260925 04:46:55.875545 14101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51a2a4c6d9504c1aabd1f8e9abfe87bb [sys.catalog]: This master's current role is: LEADER
I20260925 04:46:55.875264 14102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51a2a4c6d9504c1aabd1f8e9abfe87bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51a2a4c6d9504c1aabd1f8e9abfe87bb. Latest consensus state: current_term: 1 leader_uuid: "51a2a4c6d9504c1aabd1f8e9abfe87bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51a2a4c6d9504c1aabd1f8e9abfe87bb" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 40367 } } }
I20260925 04:46:55.875752 14102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51a2a4c6d9504c1aabd1f8e9abfe87bb [sys.catalog]: This master's current role is: LEADER
W20260925 04:46:55.878342 14114 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 51a2a4c6d9504c1aabd1f8e9abfe87bb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 04:46:55.878451 14114 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 04:46:55.878536 14115 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:46:55.879287 14115 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:46:55.884893 14115 catalog_manager.cc:1347] Generated new cluster ID: 74133decc52a4ffcbdfa190421993814
I20260925 04:46:55.884969 14115 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:46:55.900595 14115 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:46:55.901594 14115 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:46:55.914631 14115 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 51a2a4c6d9504c1aabd1f8e9abfe87bb: Generated new TSK 0
I20260925 04:46:55.915380 14115 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:46:55.929190 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 14027
2026-09-25T04:46:55Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1414 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:46:55 dist-test-slave-kghz krb5kdc[14122](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:46:55 dist-test-slave-kghz krb5kdc[14122](info): set up 2 sockets
Sep 25 04:46:55 dist-test-slave-kghz krb5kdc[14122](info): commencing operation
krb5kdc: starting...
W20260925 04:46:56.998590 393 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/krb5kdc/test-user.keytab.
Sep 25 04:46:57 dist-test-slave-kghz krb5kdc[14122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:46:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:46:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260925 04:46:57.125562 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:39037
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:35919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:39037
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:46:57.239539 14135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:46:57.239815 14135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:46:57.239857 14135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:46:57.243916 14135 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:46:57.243976 14135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:46:57.243999 14135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:46:57.244017 14135 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:46:57.244032 14135 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:46:57.249449 14135 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:35919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:39037
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:39037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14135
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:46:57.250571 14135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:46:57.251542 14135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:46:57.260075 14143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:46:57.260075 14140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:46:57.260056 14141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:46:57.260382 14135 server_base.cc:1034] running on GCE node
I20260925 04:46:57.260960 14135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:46:57.262337 14135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:46:57.263507 14135 hybrid_clock.cc:648] HybridClock initialized: now 1790311617263479 us; error 41 us; skew 500 ppm
Sep 25 04:46:57 dist-test-slave-kghz krb5kdc[14122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311617, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:46:57.266284 14135 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260925 04:46:57.267218 14135 webserver.cc:460] Webserver started at http://127.0.98.126:42833/ using document root <none> and password file <none>
I20260925 04:46:57.267782 14135 fs_manager.cc:359] Metadata directory not provided
I20260925 04:46:57.267875 14135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:46:57.268106 14135 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:46:57.270183 14135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/master-0/data/instance:
uuid: "ae3eeb992c414679aa64a6b0a2d2c0aa"
format_stamp: "Formatted at 2026-09-25 04:46:57 on dist-test-slave-kghz"
server_key: "a89214cbd07d8a9a59f2f8c1fb80a4d8"
server_key_iv: "6fd0c1701b7be61f6b6ffb724fe301ca"
server_key_version: "encryptionkey@0"
I20260925 04:46:57.270807 14135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/master-0/wal/instance:
uuid: "ae3eeb992c414679aa64a6b0a2d2c0aa"
format_stamp: "Formatted at 2026-09-25 04:46:57 on dist-test-slave-kghz"
server_key: "a89214cbd07d8a9a59f2f8c1fb80a4d8"
server_key_iv: "6fd0c1701b7be61f6b6ffb724fe301ca"
server_key_version: "encryptionkey@0"
I20260925 04:46:57.274542 14135 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 04:46:57.277019 14150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:46:57.278318 14135 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 04:46:57.278466 14135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/master-0/wal
uuid: "ae3eeb992c414679aa64a6b0a2d2c0aa"
format_stamp: "Formatted at 2026-09-25 04:46:57 on dist-test-slave-kghz"
server_key: "a89214cbd07d8a9a59f2f8c1fb80a4d8"
server_key_iv: "6fd0c1701b7be61f6b6ffb724fe301ca"
server_key_version: "encryptionkey@0"
I20260925 04:46:57.278584 14135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:46:57.296571 14135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:46:57.299630 14135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:46:57.299836 14135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:46:57.306814 14135 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:39037
I20260925 04:46:57.307103 14202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:39037 every 8 connection(s)
I20260925 04:46:57.307880 14135 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
I20260925 04:46:57.310650 14203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:46:57.311959 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 14135
I20260925 04:46:57.312091 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/master-0/wal/instance
I20260925 04:46:57.312371 393 external_mini_cluster.cc:1428] Setting key 82b83ee1fa57a0b073d8d2ebd1aa8ef2
I20260925 04:46:57.316694 14203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae3eeb992c414679aa64a6b0a2d2c0aa: Bootstrap starting.
I20260925 04:46:57.319034 14203 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae3eeb992c414679aa64a6b0a2d2c0aa: Neither blocks nor log segments found. Creating new log.
I20260925 04:46:57.319782 14203 log.cc:824] T 00000000000000000000000000000000 P ae3eeb992c414679aa64a6b0a2d2c0aa: Log is configured to *not* fsync() on all Append() calls
Sep 25 04:46:57 dist-test-slave-kghz krb5kdc[14122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:46:57.322332 14203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae3eeb992c414679aa64a6b0a2d2c0aa: No bootstrap required, opened a new log
I20260925 04:46:57.327243 14203 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae3eeb992c414679aa64a6b0a2d2c0aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae3eeb992c414679aa64a6b0a2d2c0aa" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39037 } }
I20260925 04:46:57.327512 14203 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae3eeb992c414679aa64a6b0a2d2c0aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:46:57.327556 14203 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae3eeb992c414679aa64a6b0a2d2c0aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae3eeb992c414679aa64a6b0a2d2c0aa, State: Initialized, Role: FOLLOWER
I20260925 04:46:57.328025 14206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:57.313845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50882 (local address 127.0.98.126:39037)
0925 04:46:57.314358 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0925 04:46:57.314375 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 04:46:57.314413 (+ 38us) server_negotiation.cc:407] Connection header received
0925 04:46:57.315160 (+ 747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:46:57.315177 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:46:57.315500 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:46:57.315840 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:46:57.316670 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:57.317723 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:57.318695 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:57.319210 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:57.319483 (+ 273us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:57.322648 (+ 3165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:46:57.322654 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:46:57.322671 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:46:57.322706 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:46:57.324688 (+ 1982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:46:57.325135 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:46:57.325141 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:46:57.325148 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:46:57.325285 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:46:57.325574 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:46:57.325577 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:46:57.325579 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:46:57.326103 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:46:57.326332 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0925 04:46:57.326764 (+ 432us) server_negotiation.cc:299] Negotiation successful
0925 04:46:57.327059 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":134,"threads_started":1}
I20260925 04:46:57.328022 14203 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae3eeb992c414679aa64a6b0a2d2c0aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae3eeb992c414679aa64a6b0a2d2c0aa" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39037 } }
I20260925 04:46:57.328225 14203 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae3eeb992c414679aa64a6b0a2d2c0aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:46:57.328298 14203 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae3eeb992c414679aa64a6b0a2d2c0aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:46:57.328393 14203 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae3eeb992c414679aa64a6b0a2d2c0aa [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:46:57.329478 14203 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae3eeb992c414679aa64a6b0a2d2c0aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae3eeb992c414679aa64a6b0a2d2c0aa" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39037 } }
I20260925 04:46:57.329823 14203 leader_election.cc:302] T 00000000000000000000000000000000 P ae3eeb992c414679aa64a6b0a2d2c0aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae3eeb992c414679aa64a6b0a2d2c0aa; no voters:
I20260925 04:46:57.330161 14203 leader_election.cc:288] T 00000000000000000000000000000000 P ae3eeb992c414679aa64a6b0a2d2c0aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:46:57.330288 14208 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae3eeb992c414679aa64a6b0a2d2c0aa [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:46:57.330502 14208 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae3eeb992c414679aa64a6b0a2d2c0aa [term 1 LEADER]: Becoming Leader. State: Replica: ae3eeb992c414679aa64a6b0a2d2c0aa, State: Running, Role: LEADER
I20260925 04:46:57.330842 14208 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae3eeb992c414679aa64a6b0a2d2c0aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae3eeb992c414679aa64a6b0a2d2c0aa" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39037 } }
I20260925 04:46:57.331511 14203 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae3eeb992c414679aa64a6b0a2d2c0aa [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:46:57.332756 14210 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae3eeb992c414679aa64a6b0a2d2c0aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae3eeb992c414679aa64a6b0a2d2c0aa. Latest consensus state: current_term: 1 leader_uuid: "ae3eeb992c414679aa64a6b0a2d2c0aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae3eeb992c414679aa64a6b0a2d2c0aa" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39037 } } }
I20260925 04:46:57.332897 14210 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae3eeb992c414679aa64a6b0a2d2c0aa [sys.catalog]: This master's current role is: LEADER
I20260925 04:46:57.333246 14209 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae3eeb992c414679aa64a6b0a2d2c0aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae3eeb992c414679aa64a6b0a2d2c0aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae3eeb992c414679aa64a6b0a2d2c0aa" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39037 } } }
I20260925 04:46:57.333381 14209 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae3eeb992c414679aa64a6b0a2d2c0aa [sys.catalog]: This master's current role is: LEADER
W20260925 04:46:57.336496 14223 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ae3eeb992c414679aa64a6b0a2d2c0aa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 04:46:57.336563 14223 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 04:46:57.336647 14217 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:46:57.337317 14217 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:46:57.343003 14217 catalog_manager.cc:1347] Generated new cluster ID: bf5babe13d3e42aba7c2aed4edce0baf
I20260925 04:46:57.343071 14217 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:46:57.385185 14217 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:46:57.386617 14217 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:46:57.395076 14217 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae3eeb992c414679aa64a6b0a2d2c0aa: Generated new TSK 0
I20260925 04:46:57.395893 14217 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/krb5kdc/kudu_127.0.98.65.keytab.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/krb5kdc/kudu_127.0.98.65.keytab.
I20260925 04:46:57.426164 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:0
--local_ip_for_outbound_sockets=127.0.98.65
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:39037
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:35919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:46:57.539400 14229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:46:57.539657 14229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:46:57.539700 14229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:46:57.543774 14229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:46:57.543833 14229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:46:57.543891 14229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260925 04:46:57.549127 14229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:35919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:39037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14229
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:46:57.550148 14229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:46:57.551177 14229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:46:57.560796 14237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:46:57.560796 14234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:46:57.560814 14235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:46:57.560832 14229 server_base.cc:1034] running on GCE node
I20260925 04:46:57.561448 14229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:46:57.562083 14229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:46:57.563265 14229 hybrid_clock.cc:648] HybridClock initialized: now 1790311617563252 us; error 51 us; skew 500 ppm
Sep 25 04:46:57 dist-test-slave-kghz krb5kdc[14122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311617, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:46:57.565958 14229 init.cc:377] Logged in from keytab as kudu/127.0.98.65@KRBTEST.COM (short username kudu)
I20260925 04:46:57.566872 14229 webserver.cc:460] Webserver started at http://127.0.98.65:36377/ using document root <none> and password file <none>
I20260925 04:46:57.567425 14229 fs_manager.cc:359] Metadata directory not provided
I20260925 04:46:57.567513 14229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:46:57.567762 14229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:46:57.570053 14229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-0/data/instance:
uuid: "38d6b9d0fc1649c0bc7c28d0f84bad9e"
format_stamp: "Formatted at 2026-09-25 04:46:57 on dist-test-slave-kghz"
server_key: "f0af70068975869c0b1e1a0b0030da27"
server_key_iv: "9926876c9c9692f06e54602eeec8911d"
server_key_version: "encryptionkey@0"
I20260925 04:46:57.570688 14229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-0/wal/instance:
uuid: "38d6b9d0fc1649c0bc7c28d0f84bad9e"
format_stamp: "Formatted at 2026-09-25 04:46:57 on dist-test-slave-kghz"
server_key: "f0af70068975869c0b1e1a0b0030da27"
server_key_iv: "9926876c9c9692f06e54602eeec8911d"
server_key_version: "encryptionkey@0"
I20260925 04:46:57.574375 14229 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 04:46:57.576705 14244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:46:57.577996 14229 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 04:46:57.578135 14229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-0/wal
uuid: "38d6b9d0fc1649c0bc7c28d0f84bad9e"
format_stamp: "Formatted at 2026-09-25 04:46:57 on dist-test-slave-kghz"
server_key: "f0af70068975869c0b1e1a0b0030da27"
server_key_iv: "9926876c9c9692f06e54602eeec8911d"
server_key_version: "encryptionkey@0"
I20260925 04:46:57.578241 14229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:46:57.591508 14229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:46:57.594766 14229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:46:57.594977 14229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:46:57.595551 14229 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:46:57.596345 14229 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:46:57.596415 14229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:46:57.596482 14229 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:46:57.596535 14229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:46:57.605014 14229 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:41417
I20260925 04:46:57.605039 14357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:41417 every 8 connection(s)
I20260925 04:46:57.605890 14229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
I20260925 04:46:57.612640 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 14229
I20260925 04:46:57.612778 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-0/wal/instance
I20260925 04:46:57.613070 393 external_mini_cluster.cc:1428] Setting key da855a2ca35facb6213430212a1af00d
Sep 25 04:46:57 dist-test-slave-kghz krb5kdc[14122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311617, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:46:57.620626 14206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:57.607658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:55827 (local address 127.0.98.126:39037)
0925 04:46:57.607796 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 04:46:57.607799 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:46:57.608556 (+ 757us) server_negotiation.cc:407] Connection header received
0925 04:46:57.609861 (+ 1305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:46:57.609865 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:46:57.609926 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:46:57.610029 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:46:57.611337 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:57.611894 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:57.613154 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:57.613390 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:57.613502 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:57.616411 (+ 2909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:46:57.616417 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:46:57.616422 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:46:57.616457 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:46:57.618370 (+ 1913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:46:57.618889 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:46:57.618896 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:46:57.618901 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:46:57.618995 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:46:57.619358 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:46:57.619364 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:46:57.619366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:46:57.619611 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:46:57.619721 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 04:46:57.620211 (+ 490us) server_negotiation.cc:299] Negotiation successful
0925 04:46:57.620364 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 04:46:57.621265 14360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:57.607975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39037 (local address 127.0.98.65:55827)
0925 04:46:57.608406 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0925 04:46:57.608449 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 04:46:57.609644 (+ 1195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:46:57.610208 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:46:57.610216 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:46:57.610631 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:46:57.611188 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:57.611204 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:57.612023 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:57.612026 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:57.613028 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:57.613037 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:57.613540 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:57.613545 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:57.613710 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:57.614334 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:46:57.614361 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:46:57.616197 (+ 1836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:46:57.618501 (+ 2304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:46:57.618512 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:46:57.618524 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:46:57.618786 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:46:57.619105 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:46:57.619109 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:46:57.619112 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:46:57.619259 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:46:57.619715 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:46:57.619720 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:46:57.619957 (+ 237us) client_negotiation.cc:769] Sending connection context
0925 04:46:57.620131 (+ 174us) client_negotiation.cc:240] Negotiation successful
0925 04:46:57.620390 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":116,"threads_started":1}
I20260925 04:46:57.622596 14358 heartbeater.cc:347] Connected to a master server at 127.0.98.126:39037
I20260925 04:46:57.622848 14358 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.66@KRBTEST.COM; defaulting to no policy
I20260925 04:46:57.623508 14358 heartbeater.cc:511] Master 127.0.98.126:39037 requested a full tablet report, sending...
I20260925 04:46:57.625118 14167 ts_manager.cc:194] Registered new tserver with Master: 38d6b9d0fc1649c0bc7c28d0f84bad9e (127.0.98.65:41417)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.66@KRBTEST.COM" created.
I20260925 04:46:57.626581 14167 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.65@KRBTEST.COM'} at 127.0.98.65:55827
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/krb5kdc/kudu_127.0.98.66.keytab.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/krb5kdc/kudu_127.0.98.66.keytab.
I20260925 04:46:57.643407 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:0
--local_ip_for_outbound_sockets=127.0.98.66
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:39037
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:35919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:46:57.753711 14363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:46:57.753971 14363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:46:57.754014 14363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:46:57.758108 14363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:46:57.758167 14363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:46:57.758255 14363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260925 04:46:57.763628 14363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:35919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:39037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:46:57.764739 14363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:46:57.765806 14363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:46:57.775777 14371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:46:57.775785 14369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:46:57.775785 14368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:46:57.775835 14363 server_base.cc:1034] running on GCE node
I20260925 04:46:57.776386 14363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:46:57.777098 14363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:46:57.778311 14363 hybrid_clock.cc:648] HybridClock initialized: now 1790311617778293 us; error 38 us; skew 500 ppm
Sep 25 04:46:57 dist-test-slave-kghz krb5kdc[14122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311617, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:46:57.781044 14363 init.cc:377] Logged in from keytab as kudu/127.0.98.66@KRBTEST.COM (short username kudu)
I20260925 04:46:57.781929 14363 webserver.cc:460] Webserver started at http://127.0.98.66:33731/ using document root <none> and password file <none>
I20260925 04:46:57.782613 14363 fs_manager.cc:359] Metadata directory not provided
I20260925 04:46:57.782704 14363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:46:57.782913 14363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:46:57.784936 14363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-1/data/instance:
uuid: "6f3e51d22a194e7db99512a63ba51e4f"
format_stamp: "Formatted at 2026-09-25 04:46:57 on dist-test-slave-kghz"
server_key: "e91bff69d701fad33b0c9d31d1e65f4d"
server_key_iv: "b250dfc98c43168455b1f65fa6b51e7b"
server_key_version: "encryptionkey@0"
I20260925 04:46:57.785491 14363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-1/wal/instance:
uuid: "6f3e51d22a194e7db99512a63ba51e4f"
format_stamp: "Formatted at 2026-09-25 04:46:57 on dist-test-slave-kghz"
server_key: "e91bff69d701fad33b0c9d31d1e65f4d"
server_key_iv: "b250dfc98c43168455b1f65fa6b51e7b"
server_key_version: "encryptionkey@0"
I20260925 04:46:57.789148 14363 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 04:46:57.791692 14378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:46:57.792954 14363 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 04:46:57.793092 14363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-1/wal
uuid: "6f3e51d22a194e7db99512a63ba51e4f"
format_stamp: "Formatted at 2026-09-25 04:46:57 on dist-test-slave-kghz"
server_key: "e91bff69d701fad33b0c9d31d1e65f4d"
server_key_iv: "b250dfc98c43168455b1f65fa6b51e7b"
server_key_version: "encryptionkey@0"
I20260925 04:46:57.793206 14363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:46:57.811012 14363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:46:57.814214 14363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:46:57.814469 14363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:46:57.815064 14363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:46:57.815972 14363 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:46:57.816049 14363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:46:57.816108 14363 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:46:57.816162 14363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:46:57.824736 14363 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:44125
I20260925 04:46:57.824759 14491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:44125 every 8 connection(s)
I20260925 04:46:57.825623 14363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
I20260925 04:46:57.830003 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 14363
I20260925 04:46:57.830101 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-1/wal/instance
I20260925 04:46:57.830448 393 external_mini_cluster.cc:1428] Setting key c331d543fd2bd0f91126b71bfbcc7567
Sep 25 04:46:57 dist-test-slave-kghz krb5kdc[14122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311617, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:46:57.840623 14206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:57.827416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:45659 (local address 127.0.98.126:39037)
0925 04:46:57.827559 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 04:46:57.827563 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:46:57.828335 (+ 772us) server_negotiation.cc:407] Connection header received
0925 04:46:57.829633 (+ 1298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:46:57.829637 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:46:57.829695 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:46:57.829804 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:46:57.831307 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:57.832003 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:57.833382 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:57.833624 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:57.833757 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:57.836725 (+ 2968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:46:57.836728 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:46:57.836731 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:46:57.836761 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:46:57.838518 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:46:57.839019 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:46:57.839024 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:46:57.839028 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:46:57.839122 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:46:57.839441 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:46:57.839444 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:46:57.839446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:46:57.839700 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:46:57.839813 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 04:46:57.840286 (+ 473us) server_negotiation.cc:299] Negotiation successful
0925 04:46:57.840428 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
WARNING: no policy specified for kudu/127.0.98.67@KRBTEST.COM; defaulting to no policy
I20260925 04:46:57.841336 14494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:57.827761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39037 (local address 127.0.98.66:45659)
0925 04:46:57.828190 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0925 04:46:57.828232 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 04:46:57.829420 (+ 1188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:46:57.829992 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:46:57.829998 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:46:57.830516 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:46:57.831143 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:57.831180 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:57.832167 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:57.832171 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:57.833255 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:57.833264 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:57.833797 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:57.833801 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:57.834020 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:57.834684 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:46:57.834701 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:46:57.836554 (+ 1853us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:46:57.838643 (+ 2089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:46:57.838648 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:46:57.838661 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:46:57.838913 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:46:57.839216 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:46:57.839218 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:46:57.839219 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:46:57.839344 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:46:57.839804 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:46:57.839809 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:46:57.840036 (+ 227us) client_negotiation.cc:769] Sending connection context
0925 04:46:57.840194 (+ 158us) client_negotiation.cc:240] Negotiation successful
0925 04:46:57.840441 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":115,"threads_started":1}
I20260925 04:46:57.842556 14492 heartbeater.cc:347] Connected to a master server at 127.0.98.126:39037
I20260925 04:46:57.842810 14492 heartbeater.cc:464] Registering TS with master...
I20260925 04:46:57.843353 14492 heartbeater.cc:511] Master 127.0.98.126:39037 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.67@KRBTEST.COM" created.
I20260925 04:46:57.844480 14167 ts_manager.cc:194] Registered new tserver with Master: 6f3e51d22a194e7db99512a63ba51e4f (127.0.98.66:44125)
I20260925 04:46:57.845067 14167 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.66@KRBTEST.COM'} at 127.0.98.66:45659
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/krb5kdc/kudu_127.0.98.67.keytab.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/krb5kdc/kudu_127.0.98.67.keytab.
I20260925 04:46:57.861596 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:0
--local_ip_for_outbound_sockets=127.0.98.67
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:39037
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:35919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:46:57.976279 14497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:46:57.976519 14497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:46:57.976599 14497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:46:57.980654 14497 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:46:57.980724 14497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:46:57.980818 14497 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260925 04:46:57.986465 14497 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:35919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:39037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14497
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:46:57.987498 14497 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:46:57.988505 14497 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:46:57.998131 14505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:46:57.998219 14502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:46:57.998198 14497 server_base.cc:1034] running on GCE node
W20260925 04:46:57.998201 14503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:46:57.998777 14497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:46:57.999396 14497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:46:58.000573 14497 hybrid_clock.cc:648] HybridClock initialized: now 1790311618000542 us; error 46 us; skew 500 ppm
Sep 25 04:46:58 dist-test-slave-kghz krb5kdc[14122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311618, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:46:58.003304 14497 init.cc:377] Logged in from keytab as kudu/127.0.98.67@KRBTEST.COM (short username kudu)
I20260925 04:46:58.004165 14497 webserver.cc:460] Webserver started at http://127.0.98.67:39533/ using document root <none> and password file <none>
I20260925 04:46:58.004711 14497 fs_manager.cc:359] Metadata directory not provided
I20260925 04:46:58.004770 14497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:46:58.005007 14497 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:46:58.007066 14497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-2/data/instance:
uuid: "91628bf78d7a4e0b909f85c5b6af76a4"
format_stamp: "Formatted at 2026-09-25 04:46:58 on dist-test-slave-kghz"
server_key: "91f46b945b0026e354cb60518ec43780"
server_key_iv: "e35f2d4b8d4c010ba7b881e9440673b5"
server_key_version: "encryptionkey@0"
I20260925 04:46:58.007615 14497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-2/wal/instance:
uuid: "91628bf78d7a4e0b909f85c5b6af76a4"
format_stamp: "Formatted at 2026-09-25 04:46:58 on dist-test-slave-kghz"
server_key: "91f46b945b0026e354cb60518ec43780"
server_key_iv: "e35f2d4b8d4c010ba7b881e9440673b5"
server_key_version: "encryptionkey@0"
I20260925 04:46:58.011206 14497 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 04:46:58.013621 14512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:46:58.014895 14497 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 04:46:58.015013 14497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-2/wal
uuid: "91628bf78d7a4e0b909f85c5b6af76a4"
format_stamp: "Formatted at 2026-09-25 04:46:58 on dist-test-slave-kghz"
server_key: "91f46b945b0026e354cb60518ec43780"
server_key_iv: "e35f2d4b8d4c010ba7b881e9440673b5"
server_key_version: "encryptionkey@0"
I20260925 04:46:58.015111 14497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:46:58.026427 14497 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:46:58.029310 14497 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:46:58.029524 14497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:46:58.030148 14497 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:46:58.031013 14497 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:46:58.031138 14497 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:46:58.031209 14497 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:46:58.031258 14497 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:46:58.041441 14497 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:43485
I20260925 04:46:58.041468 14625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:43485 every 8 connection(s)
I20260925 04:46:58.042413 14497 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
I20260925 04:46:58.048125 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 14497
I20260925 04:46:58.048259 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-2/wal/instance
I20260925 04:46:58.048568 393 external_mini_cluster.cc:1428] Setting key bbde41be712a0cc97ee14a7ba4ee1daa
Sep 25 04:46:58 dist-test-slave-kghz krb5kdc[14122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311618, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:46:58.056141 14206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:58.044198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:49131 (local address 127.0.98.126:39037)
0925 04:46:58.044366 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 04:46:58.044372 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:46:58.045221 (+ 849us) server_negotiation.cc:407] Connection header received
0925 04:46:58.046512 (+ 1291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:46:58.046515 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:46:58.046560 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:46:58.046638 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:46:58.047805 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:58.048423 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:58.049842 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:58.050145 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:58.050293 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:58.052788 (+ 2495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:46:58.052791 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:46:58.052793 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:46:58.052818 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:46:58.054357 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:46:58.054850 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:46:58.054853 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:46:58.054854 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:46:58.054929 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:46:58.055224 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:46:58.055227 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:46:58.055228 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:46:58.055392 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:46:58.055463 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 04:46:58.055926 (+ 463us) server_negotiation.cc:299] Negotiation successful
0925 04:46:58.056018 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 04:46:58.056998 14628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:58.044614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39037 (local address 127.0.98.67:49131)
0925 04:46:58.045067 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0925 04:46:58.045108 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 04:46:58.046299 (+ 1191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:46:58.046786 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:46:58.046792 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:46:58.047157 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:46:58.047666 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:58.047680 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:58.048549 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:58.048553 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:58.049723 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:58.049733 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:58.050334 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:58.050338 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:58.050479 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:58.051134 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:46:58.051151 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:46:58.052622 (+ 1471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:46:58.054468 (+ 1846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:46:58.054474 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:46:58.054486 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:46:58.054756 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:46:58.055004 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:46:58.055006 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:46:58.055007 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:46:58.055148 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:46:58.055472 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:46:58.055477 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:46:58.055729 (+ 252us) client_negotiation.cc:769] Sending connection context
0925 04:46:58.055882 (+ 153us) client_negotiation.cc:240] Negotiation successful
0925 04:46:58.056136 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":115,"threads_started":1}
I20260925 04:46:58.058167 14626 heartbeater.cc:347] Connected to a master server at 127.0.98.126:39037
I20260925 04:46:58.058429 14626 heartbeater.cc:464] Registering TS with master...
I20260925 04:46:58.058877 14626 heartbeater.cc:511] Master 127.0.98.126:39037 requested a full tablet report, sending...
I20260925 04:46:58.059867 14167 ts_manager.cc:194] Registered new tserver with Master: 91628bf78d7a4e0b909f85c5b6af76a4 (127.0.98.67:43485)
I20260925 04:46:58.060446 14167 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.67@KRBTEST.COM'} at 127.0.98.67:49131
I20260925 04:46:58.062645 393 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 04:46:58.072259 14206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:58.064589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50894 (local address 127.0.98.126:39037)
0925 04:46:58.064740 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 04:46:58.064745 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:46:58.064909 (+ 164us) server_negotiation.cc:407] Connection header received
0925 04:46:58.064998 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:46:58.065000 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:46:58.065038 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:46:58.065102 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:46:58.065863 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:58.066797 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:58.067624 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:58.067826 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:58.067906 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:58.069251 (+ 1345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:46:58.069254 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:46:58.069255 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:46:58.069275 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:46:58.070538 (+ 1263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:46:58.070970 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:46:58.070973 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:46:58.070974 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:46:58.071066 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:46:58.071378 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:46:58.071380 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:46:58.071381 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:46:58.071626 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:46:58.071732 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 04:46:58.071943 (+ 211us) server_negotiation.cc:299] Negotiation successful
0925 04:46:58.072036 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 04:46:58.075270 14167 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50894:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:46:58.078318 14167 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:46:58.092887 14637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:58.086504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:44125 (local address 127.0.0.1:41546)
0925 04:46:58.086848 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0925 04:46:58.086912 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0925 04:46:58.087171 (+ 259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:46:58.088498 (+ 1327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:46:58.088505 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:46:58.088531 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:46:58.088674 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:58.088682 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:58.089885 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:58.089888 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:58.091210 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:58.091220 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:58.092311 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:58.092315 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:58.092442 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:58.092491 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 04:46:58.092597 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 04:46:58.092721 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"mutex_wait_us":4,"thread_start_us":81,"threads_started":1}
I20260925 04:46:58.092940 14639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:58.086701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:41417 (local address 127.0.0.1:36746)
0925 04:46:58.087389 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0925 04:46:58.087404 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:46:58.087507 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:46:58.089064 (+ 1557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:46:58.089067 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:46:58.089083 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:46:58.089175 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:58.089181 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:58.090525 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:58.090528 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:58.091625 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:58.091634 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:58.092514 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:58.092517 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:58.092623 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:58.092656 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 04:46:58.092723 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 04:46:58.092793 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"mutex_wait_us":39,"thread_start_us":105,"threads_started":1}
I20260925 04:46:58.093117 14642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:58.087497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56886 (local address 127.0.98.67:43485)
0925 04:46:58.088368 (+ 871us) server_negotiation.cc:206] Beginning negotiation
0925 04:46:58.088381 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 04:46:58.088405 (+ 24us) server_negotiation.cc:407] Connection header received
0925 04:46:58.088483 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:46:58.088489 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:46:58.088693 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:46:58.088864 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:46:58.089085 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:58.090301 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:58.091814 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:58.092610 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:58.092723 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:58.092840 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 04:46:58.092893 (+ 53us) server_negotiation.cc:299] Negotiation successful
0925 04:46:58.092987 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":769,"thread_start_us":165,"threads_started":1}
I20260925 04:46:58.093294 14638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:58.086909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:43485 (local address 127.0.0.1:56886)
0925 04:46:58.087732 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0925 04:46:58.087744 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 04:46:58.087854 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:46:58.088840 (+ 986us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:46:58.088842 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:46:58.088859 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:46:58.088952 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:58.088958 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:58.090450 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:58.090454 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:58.091669 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:58.091678 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:58.092711 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:58.092714 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:58.092790 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:58.092811 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 04:46:58.092869 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 04:46:58.093106 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":755,"mutex_wait_us":42,"thread_start_us":88,"threads_started":1}
I20260925 04:46:58.093513 14640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:58.087157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41546 (local address 127.0.98.66:44125)
0925 04:46:58.087724 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0925 04:46:58.087736 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 04:46:58.087763 (+ 27us) server_negotiation.cc:407] Connection header received
0925 04:46:58.087854 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:46:58.087862 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:46:58.088100 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:46:58.088273 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:46:58.088811 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:58.089754 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:58.091351 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:58.092179 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:58.092290 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:58.092432 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 04:46:58.093234 (+ 802us) server_negotiation.cc:299] Negotiation successful
0925 04:46:58.093340 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":655,"threads_started":1}
I20260925 04:46:58.093757 14641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:58.087266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36746 (local address 127.0.98.65:41417)
0925 04:46:58.088591 (+ 1325us) server_negotiation.cc:206] Beginning negotiation
0925 04:46:58.088605 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 04:46:58.088625 (+ 20us) server_negotiation.cc:407] Connection header received
0925 04:46:58.088697 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:46:58.088704 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:46:58.088915 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:46:58.089070 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:46:58.089270 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:58.090385 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:58.091750 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:58.092376 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:58.092476 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:58.092608 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 04:46:58.093489 (+ 881us) server_negotiation.cc:299] Negotiation successful
0925 04:46:58.093603 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1225,"thread_start_us":78,"threads_started":1}
I20260925 04:46:58.096465 14560 tablet_service.cc:1467] Processing CreateTablet for tablet a2f73a46ecba4ae097f51f8078e83f33 (DEFAULT_TABLE table=test-table [id=4cdb6641d9c7432d8b5275737dbc9080]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:46:58.096468 14426 tablet_service.cc:1467] Processing CreateTablet for tablet a2f73a46ecba4ae097f51f8078e83f33 (DEFAULT_TABLE table=test-table [id=4cdb6641d9c7432d8b5275737dbc9080]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:46:58.097640 14560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2f73a46ecba4ae097f51f8078e83f33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:46:58.097637 14426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2f73a46ecba4ae097f51f8078e83f33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:46:58.104516 14292 tablet_service.cc:1467] Processing CreateTablet for tablet a2f73a46ecba4ae097f51f8078e83f33 (DEFAULT_TABLE table=test-table [id=4cdb6641d9c7432d8b5275737dbc9080]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:46:58.105106 14643 tablet_bootstrap.cc:492] T a2f73a46ecba4ae097f51f8078e83f33 P 6f3e51d22a194e7db99512a63ba51e4f: Bootstrap starting.
I20260925 04:46:58.105648 14292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2f73a46ecba4ae097f51f8078e83f33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:46:58.106431 14644 tablet_bootstrap.cc:492] T a2f73a46ecba4ae097f51f8078e83f33 P 91628bf78d7a4e0b909f85c5b6af76a4: Bootstrap starting.
I20260925 04:46:58.107959 14643 tablet_bootstrap.cc:654] T a2f73a46ecba4ae097f51f8078e83f33 P 6f3e51d22a194e7db99512a63ba51e4f: Neither blocks nor log segments found. Creating new log.
I20260925 04:46:58.108592 14644 tablet_bootstrap.cc:654] T a2f73a46ecba4ae097f51f8078e83f33 P 91628bf78d7a4e0b909f85c5b6af76a4: Neither blocks nor log segments found. Creating new log.
I20260925 04:46:58.109062 14643 log.cc:824] T a2f73a46ecba4ae097f51f8078e83f33 P 6f3e51d22a194e7db99512a63ba51e4f: Log is configured to *not* fsync() on all Append() calls
I20260925 04:46:58.109764 14644 log.cc:824] T a2f73a46ecba4ae097f51f8078e83f33 P 91628bf78d7a4e0b909f85c5b6af76a4: Log is configured to *not* fsync() on all Append() calls
I20260925 04:46:58.111289 14643 tablet_bootstrap.cc:492] T a2f73a46ecba4ae097f51f8078e83f33 P 6f3e51d22a194e7db99512a63ba51e4f: No bootstrap required, opened a new log
I20260925 04:46:58.111506 14643 ts_tablet_manager.cc:1397] T a2f73a46ecba4ae097f51f8078e83f33 P 6f3e51d22a194e7db99512a63ba51e4f: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 04:46:58.111690 14644 tablet_bootstrap.cc:492] T a2f73a46ecba4ae097f51f8078e83f33 P 91628bf78d7a4e0b909f85c5b6af76a4: No bootstrap required, opened a new log
I20260925 04:46:58.111864 14644 ts_tablet_manager.cc:1397] T a2f73a46ecba4ae097f51f8078e83f33 P 91628bf78d7a4e0b909f85c5b6af76a4: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 04:46:58.112159 14647 tablet_bootstrap.cc:492] T a2f73a46ecba4ae097f51f8078e83f33 P 38d6b9d0fc1649c0bc7c28d0f84bad9e: Bootstrap starting.
I20260925 04:46:58.114004 14647 tablet_bootstrap.cc:654] T a2f73a46ecba4ae097f51f8078e83f33 P 38d6b9d0fc1649c0bc7c28d0f84bad9e: Neither blocks nor log segments found. Creating new log.
I20260925 04:46:58.115123 14647 log.cc:824] T a2f73a46ecba4ae097f51f8078e83f33 P 38d6b9d0fc1649c0bc7c28d0f84bad9e: Log is configured to *not* fsync() on all Append() calls
I20260925 04:46:58.115528 14643 raft_consensus.cc:348] T a2f73a46ecba4ae097f51f8078e83f33 P 6f3e51d22a194e7db99512a63ba51e4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f3e51d22a194e7db99512a63ba51e4f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 44125 } } peers { permanent_uuid: "38d6b9d0fc1649c0bc7c28d0f84bad9e" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41417 } } peers { permanent_uuid: "91628bf78d7a4e0b909f85c5b6af76a4" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43485 } }
I20260925 04:46:58.115528 14644 raft_consensus.cc:348] T a2f73a46ecba4ae097f51f8078e83f33 P 91628bf78d7a4e0b909f85c5b6af76a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f3e51d22a194e7db99512a63ba51e4f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 44125 } } peers { permanent_uuid: "38d6b9d0fc1649c0bc7c28d0f84bad9e" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41417 } } peers { permanent_uuid: "91628bf78d7a4e0b909f85c5b6af76a4" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43485 } }
I20260925 04:46:58.115813 14643 raft_consensus.cc:374] T a2f73a46ecba4ae097f51f8078e83f33 P 6f3e51d22a194e7db99512a63ba51e4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:46:58.115813 14644 raft_consensus.cc:374] T a2f73a46ecba4ae097f51f8078e83f33 P 91628bf78d7a4e0b909f85c5b6af76a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:46:58.115864 14643 raft_consensus.cc:729] T a2f73a46ecba4ae097f51f8078e83f33 P 6f3e51d22a194e7db99512a63ba51e4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f3e51d22a194e7db99512a63ba51e4f, State: Initialized, Role: FOLLOWER
I20260925 04:46:58.115864 14644 raft_consensus.cc:729] T a2f73a46ecba4ae097f51f8078e83f33 P 91628bf78d7a4e0b909f85c5b6af76a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91628bf78d7a4e0b909f85c5b6af76a4, State: Initialized, Role: FOLLOWER
I20260925 04:46:58.116369 14644 consensus_queue.cc:260] T a2f73a46ecba4ae097f51f8078e83f33 P 91628bf78d7a4e0b909f85c5b6af76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f3e51d22a194e7db99512a63ba51e4f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 44125 } } peers { permanent_uuid: "38d6b9d0fc1649c0bc7c28d0f84bad9e" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41417 } } peers { permanent_uuid: "91628bf78d7a4e0b909f85c5b6af76a4" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43485 } }
I20260925 04:46:58.116369 14643 consensus_queue.cc:260] T a2f73a46ecba4ae097f51f8078e83f33 P 6f3e51d22a194e7db99512a63ba51e4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f3e51d22a194e7db99512a63ba51e4f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 44125 } } peers { permanent_uuid: "38d6b9d0fc1649c0bc7c28d0f84bad9e" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41417 } } peers { permanent_uuid: "91628bf78d7a4e0b909f85c5b6af76a4" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43485 } }
I20260925 04:46:58.117079 14647 tablet_bootstrap.cc:492] T a2f73a46ecba4ae097f51f8078e83f33 P 38d6b9d0fc1649c0bc7c28d0f84bad9e: No bootstrap required, opened a new log
I20260925 04:46:58.117281 14647 ts_tablet_manager.cc:1397] T a2f73a46ecba4ae097f51f8078e83f33 P 38d6b9d0fc1649c0bc7c28d0f84bad9e: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260925 04:46:58.117398 14644 ts_tablet_manager.cc:1428] T a2f73a46ecba4ae097f51f8078e83f33 P 91628bf78d7a4e0b909f85c5b6af76a4: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260925 04:46:58.117372 14492 heartbeater.cc:503] Master 127.0.98.126:39037 was elected leader, sending a full tablet report...
I20260925 04:46:58.117406 14643 ts_tablet_manager.cc:1428] T a2f73a46ecba4ae097f51f8078e83f33 P 6f3e51d22a194e7db99512a63ba51e4f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 04:46:58.117879 14626 heartbeater.cc:503] Master 127.0.98.126:39037 was elected leader, sending a full tablet report...
I20260925 04:46:58.121219 14647 raft_consensus.cc:348] T a2f73a46ecba4ae097f51f8078e83f33 P 38d6b9d0fc1649c0bc7c28d0f84bad9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f3e51d22a194e7db99512a63ba51e4f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 44125 } } peers { permanent_uuid: "38d6b9d0fc1649c0bc7c28d0f84bad9e" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41417 } } peers { permanent_uuid: "91628bf78d7a4e0b909f85c5b6af76a4" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43485 } }
I20260925 04:46:58.121451 14647 raft_consensus.cc:374] T a2f73a46ecba4ae097f51f8078e83f33 P 38d6b9d0fc1649c0bc7c28d0f84bad9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:46:58.121513 14647 raft_consensus.cc:729] T a2f73a46ecba4ae097f51f8078e83f33 P 38d6b9d0fc1649c0bc7c28d0f84bad9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38d6b9d0fc1649c0bc7c28d0f84bad9e, State: Initialized, Role: FOLLOWER
I20260925 04:46:58.121989 14647 consensus_queue.cc:260] T a2f73a46ecba4ae097f51f8078e83f33 P 38d6b9d0fc1649c0bc7c28d0f84bad9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f3e51d22a194e7db99512a63ba51e4f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 44125 } } peers { permanent_uuid: "38d6b9d0fc1649c0bc7c28d0f84bad9e" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41417 } } peers { permanent_uuid: "91628bf78d7a4e0b909f85c5b6af76a4" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43485 } }
I20260925 04:46:58.122635 14358 heartbeater.cc:503] Master 127.0.98.126:39037 was elected leader, sending a full tablet report...
I20260925 04:46:58.122920 14647 ts_tablet_manager.cc:1428] T a2f73a46ecba4ae097f51f8078e83f33 P 38d6b9d0fc1649c0bc7c28d0f84bad9e: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
W20260925 04:46:58.293344 14627 tablet.cc:2369] T a2f73a46ecba4ae097f51f8078e83f33 P 91628bf78d7a4e0b909f85c5b6af76a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 04:46:58.326725 14493 tablet.cc:2369] T a2f73a46ecba4ae097f51f8078e83f33 P 6f3e51d22a194e7db99512a63ba51e4f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:46:58.326740 14651 raft_consensus.cc:482] T a2f73a46ecba4ae097f51f8078e83f33 P 38d6b9d0fc1649c0bc7c28d0f84bad9e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:46:58.326946 14651 raft_consensus.cc:504] T a2f73a46ecba4ae097f51f8078e83f33 P 38d6b9d0fc1649c0bc7c28d0f84bad9e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f3e51d22a194e7db99512a63ba51e4f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 44125 } } peers { permanent_uuid: "38d6b9d0fc1649c0bc7c28d0f84bad9e" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41417 } } peers { permanent_uuid: "91628bf78d7a4e0b909f85c5b6af76a4" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43485 } }
I20260925 04:46:58.328066 14651 leader_election.cc:288] T a2f73a46ecba4ae097f51f8078e83f33 P 38d6b9d0fc1649c0bc7c28d0f84bad9e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f3e51d22a194e7db99512a63ba51e4f (127.0.98.66:44125), 91628bf78d7a4e0b909f85c5b6af76a4 (127.0.98.67:43485)
I20260925 04:46:58.331985 14640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:58.328373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:34025 (local address 127.0.98.66:44125)
0925 04:46:58.328655 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0925 04:46:58.328659 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:46:58.328813 (+ 154us) server_negotiation.cc:407] Connection header received
0925 04:46:58.328964 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:46:58.328967 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:46:58.329018 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:46:58.329108 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:46:58.329401 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:58.329897 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:58.331027 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:58.331424 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:58.331536 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:58.331606 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 04:46:58.331780 (+ 174us) server_negotiation.cc:299] Negotiation successful
0925 04:46:58.331849 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190}
I20260925 04:46:58.332046 14653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:58.328300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:44125 (local address 127.0.98.65:34025)
0925 04:46:58.328724 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0925 04:46:58.328735 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 04:46:58.328857 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:46:58.329127 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:46:58.329130 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:46:58.329156 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:46:58.329292 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:58.329297 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:58.330009 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:58.330013 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:58.330903 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:58.330912 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:58.331550 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:58.331554 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:58.331652 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:58.331677 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 04:46:58.331764 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 04:46:58.331832 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"mutex_wait_us":81,"spinlock_wait_cycles":1920,"thread_start_us":137,"threads_started":1}
I20260925 04:46:58.332365 14642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:58.328387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:55263 (local address 127.0.98.67:43485)
0925 04:46:58.328544 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 04:46:58.328551 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:46:58.328715 (+ 164us) server_negotiation.cc:407] Connection header received
0925 04:46:58.328909 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:46:58.328913 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:46:58.328972 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:46:58.329061 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:46:58.329381 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:58.329941 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:58.331247 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:58.331688 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:58.331816 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:58.331878 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 04:46:58.332161 (+ 283us) server_negotiation.cc:299] Negotiation successful
0925 04:46:58.332248 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 04:46:58.332779 14652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:58.328298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:43485 (local address 127.0.98.65:55263)
0925 04:46:58.328618 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0925 04:46:58.328637 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 04:46:58.328763 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:46:58.329085 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:46:58.329088 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:46:58.329105 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:46:58.329261 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:58.329266 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:58.330058 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:58.330065 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:58.331122 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:58.331132 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:58.331818 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:58.331822 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:58.331987 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:58.332016 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 04:46:58.332534 (+ 518us) client_negotiation.cc:240] Negotiation successful
0925 04:46:58.332616 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"mutex_wait_us":3,"thread_start_us":115,"threads_started":1}
I20260925 04:46:58.333001 14446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2f73a46ecba4ae097f51f8078e83f33" candidate_uuid: "38d6b9d0fc1649c0bc7c28d0f84bad9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f3e51d22a194e7db99512a63ba51e4f" is_pre_election: true
I20260925 04:46:58.333319 14446 raft_consensus.cc:2454] T a2f73a46ecba4ae097f51f8078e83f33 P 6f3e51d22a194e7db99512a63ba51e4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38d6b9d0fc1649c0bc7c28d0f84bad9e in term 0.
I20260925 04:46:58.333391 14580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2f73a46ecba4ae097f51f8078e83f33" candidate_uuid: "38d6b9d0fc1649c0bc7c28d0f84bad9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91628bf78d7a4e0b909f85c5b6af76a4" is_pre_election: true
I20260925 04:46:58.333711 14580 raft_consensus.cc:2454] T a2f73a46ecba4ae097f51f8078e83f33 P 91628bf78d7a4e0b909f85c5b6af76a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38d6b9d0fc1649c0bc7c28d0f84bad9e in term 0.
I20260925 04:46:58.333794 14247 leader_election.cc:302] T a2f73a46ecba4ae097f51f8078e83f33 P 38d6b9d0fc1649c0bc7c28d0f84bad9e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38d6b9d0fc1649c0bc7c28d0f84bad9e, 6f3e51d22a194e7db99512a63ba51e4f; no voters:
I20260925 04:46:58.334125 14651 raft_consensus.cc:2781] T a2f73a46ecba4ae097f51f8078e83f33 P 38d6b9d0fc1649c0bc7c28d0f84bad9e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:46:58.334227 14651 raft_consensus.cc:482] T a2f73a46ecba4ae097f51f8078e83f33 P 38d6b9d0fc1649c0bc7c28d0f84bad9e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:46:58.334311 14651 raft_consensus.cc:3037] T a2f73a46ecba4ae097f51f8078e83f33 P 38d6b9d0fc1649c0bc7c28d0f84bad9e [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:46:58.335374 14651 raft_consensus.cc:504] T a2f73a46ecba4ae097f51f8078e83f33 P 38d6b9d0fc1649c0bc7c28d0f84bad9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f3e51d22a194e7db99512a63ba51e4f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 44125 } } peers { permanent_uuid: "38d6b9d0fc1649c0bc7c28d0f84bad9e" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41417 } } peers { permanent_uuid: "91628bf78d7a4e0b909f85c5b6af76a4" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43485 } }
I20260925 04:46:58.335726 14651 leader_election.cc:288] T a2f73a46ecba4ae097f51f8078e83f33 P 38d6b9d0fc1649c0bc7c28d0f84bad9e [CANDIDATE]: Term 1 election: Requested vote from peers 6f3e51d22a194e7db99512a63ba51e4f (127.0.98.66:44125), 91628bf78d7a4e0b909f85c5b6af76a4 (127.0.98.67:43485)
I20260925 04:46:58.336063 14446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2f73a46ecba4ae097f51f8078e83f33" candidate_uuid: "38d6b9d0fc1649c0bc7c28d0f84bad9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f3e51d22a194e7db99512a63ba51e4f"
I20260925 04:46:58.336077 14580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2f73a46ecba4ae097f51f8078e83f33" candidate_uuid: "38d6b9d0fc1649c0bc7c28d0f84bad9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91628bf78d7a4e0b909f85c5b6af76a4"
I20260925 04:46:58.336198 14446 raft_consensus.cc:3037] T a2f73a46ecba4ae097f51f8078e83f33 P 6f3e51d22a194e7db99512a63ba51e4f [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:46:58.336198 14580 raft_consensus.cc:3037] T a2f73a46ecba4ae097f51f8078e83f33 P 91628bf78d7a4e0b909f85c5b6af76a4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:46:58.337548 14580 raft_consensus.cc:2454] T a2f73a46ecba4ae097f51f8078e83f33 P 91628bf78d7a4e0b909f85c5b6af76a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38d6b9d0fc1649c0bc7c28d0f84bad9e in term 1.
I20260925 04:46:58.337548 14446 raft_consensus.cc:2454] T a2f73a46ecba4ae097f51f8078e83f33 P 6f3e51d22a194e7db99512a63ba51e4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38d6b9d0fc1649c0bc7c28d0f84bad9e in term 1.
I20260925 04:46:58.337809 14247 leader_election.cc:302] T a2f73a46ecba4ae097f51f8078e83f33 P 38d6b9d0fc1649c0bc7c28d0f84bad9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38d6b9d0fc1649c0bc7c28d0f84bad9e, 6f3e51d22a194e7db99512a63ba51e4f; no voters:
I20260925 04:46:58.337992 14651 raft_consensus.cc:2781] T a2f73a46ecba4ae097f51f8078e83f33 P 38d6b9d0fc1649c0bc7c28d0f84bad9e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:46:58.338233 14651 raft_consensus.cc:686] T a2f73a46ecba4ae097f51f8078e83f33 P 38d6b9d0fc1649c0bc7c28d0f84bad9e [term 1 LEADER]: Becoming Leader. State: Replica: 38d6b9d0fc1649c0bc7c28d0f84bad9e, State: Running, Role: LEADER
I20260925 04:46:58.338599 14651 consensus_queue.cc:237] T a2f73a46ecba4ae097f51f8078e83f33 P 38d6b9d0fc1649c0bc7c28d0f84bad9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f3e51d22a194e7db99512a63ba51e4f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 44125 } } peers { permanent_uuid: "38d6b9d0fc1649c0bc7c28d0f84bad9e" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41417 } } peers { permanent_uuid: "91628bf78d7a4e0b909f85c5b6af76a4" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43485 } }
I20260925 04:46:58.341681 14166 catalog_manager.cc:5495] T a2f73a46ecba4ae097f51f8078e83f33 P 38d6b9d0fc1649c0bc7c28d0f84bad9e reported cstate change: term changed from 0 to 1, leader changed from <none> to 38d6b9d0fc1649c0bc7c28d0f84bad9e (127.0.98.65). New cstate: current_term: 1 leader_uuid: "38d6b9d0fc1649c0bc7c28d0f84bad9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f3e51d22a194e7db99512a63ba51e4f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 44125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38d6b9d0fc1649c0bc7c28d0f84bad9e" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91628bf78d7a4e0b909f85c5b6af76a4" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43485 } health_report { overall_health: UNKNOWN } } }
W20260925 04:46:58.357199 14359 tablet.cc:2369] T a2f73a46ecba4ae097f51f8078e83f33 P 38d6b9d0fc1649c0bc7c28d0f84bad9e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:46:58.385504 14641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:58.381771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36760 (local address 127.0.98.65:41417)
0925 04:46:58.381908 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 04:46:58.381912 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:46:58.381974 (+ 62us) server_negotiation.cc:407] Connection header received
0925 04:46:58.382184 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:46:58.382187 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:46:58.382292 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:46:58.382412 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:46:58.382668 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:58.383262 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:58.384120 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:58.384346 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:58.384441 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:58.384697 (+ 256us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:46:58.385115 (+ 418us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:46:58.385208 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 04:46:58.385253 (+ 45us) server_negotiation.cc:299] Negotiation successful
0925 04:46:58.385302 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 04:46:58.391178 14580 raft_consensus.cc:1261] T a2f73a46ecba4ae097f51f8078e83f33 P 91628bf78d7a4e0b909f85c5b6af76a4 [term 1 FOLLOWER]: Refusing update from remote peer 38d6b9d0fc1649c0bc7c28d0f84bad9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:46:58.391179 14446 raft_consensus.cc:1261] T a2f73a46ecba4ae097f51f8078e83f33 P 6f3e51d22a194e7db99512a63ba51e4f [term 1 FOLLOWER]: Refusing update from remote peer 38d6b9d0fc1649c0bc7c28d0f84bad9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:46:58.391937 14654 consensus_queue.cc:1035] T a2f73a46ecba4ae097f51f8078e83f33 P 38d6b9d0fc1649c0bc7c28d0f84bad9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f3e51d22a194e7db99512a63ba51e4f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 44125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:46:58.392184 14651 consensus_queue.cc:1035] T a2f73a46ecba4ae097f51f8078e83f33 P 38d6b9d0fc1649c0bc7c28d0f84bad9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "91628bf78d7a4e0b909f85c5b6af76a4" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:46:58.399528 14659 mvcc.cc:205] Tried to move back new op lower bound from 7333116388923203584 to 7333116388717748224. Current Snapshot: MvccSnapshot[applied={T|T < 7333116388923203584}]
I20260925 04:46:58.402628 14658 mvcc.cc:205] Tried to move back new op lower bound from 7333116388923203584 to 7333116388717748224. Current Snapshot: MvccSnapshot[applied={T|T < 7333116388923203584 or (T in {7333116388923203584})}]
I20260925 04:46:58.406673 14166 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50894:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:46:58.406845 14166 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50894:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:46:58.409191 14166 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ae3eeb992c414679aa64a6b0a2d2c0aa: Sending DeleteTablet for 3 replicas of tablet a2f73a46ecba4ae097f51f8078e83f33
I20260925 04:46:58.409899 14426 tablet_service.cc:1514] Processing DeleteTablet for tablet a2f73a46ecba4ae097f51f8078e83f33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:46:58 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:41546
I20260925 04:46:58.409902 14292 tablet_service.cc:1514] Processing DeleteTablet for tablet a2f73a46ecba4ae097f51f8078e83f33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:46:58 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:36746
I20260925 04:46:58.410080 14560 tablet_service.cc:1514] Processing DeleteTablet for tablet a2f73a46ecba4ae097f51f8078e83f33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:46:58 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:56886
I20260925 04:46:58.410522 14666 tablet_replica.cc:330] T a2f73a46ecba4ae097f51f8078e83f33 P 6f3e51d22a194e7db99512a63ba51e4f: stopping tablet replica
I20260925 04:46:58.410553 14668 tablet_replica.cc:330] T a2f73a46ecba4ae097f51f8078e83f33 P 91628bf78d7a4e0b909f85c5b6af76a4: stopping tablet replica
I20260925 04:46:58.410817 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 14229
I20260925 04:46:58.410854 14666 raft_consensus.cc:2229] T a2f73a46ecba4ae097f51f8078e83f33 P 6f3e51d22a194e7db99512a63ba51e4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:46:58.411140 14666 raft_consensus.cc:2258] T a2f73a46ecba4ae097f51f8078e83f33 P 6f3e51d22a194e7db99512a63ba51e4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:46:58.410864 14668 raft_consensus.cc:2229] T a2f73a46ecba4ae097f51f8078e83f33 P 91628bf78d7a4e0b909f85c5b6af76a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:46:58.412029 14668 raft_consensus.cc:2258] T a2f73a46ecba4ae097f51f8078e83f33 P 91628bf78d7a4e0b909f85c5b6af76a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:46:58.412940 14666 ts_tablet_manager.cc:1905] T a2f73a46ecba4ae097f51f8078e83f33 P 6f3e51d22a194e7db99512a63ba51e4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:46:58.413573 14668 ts_tablet_manager.cc:1905] T a2f73a46ecba4ae097f51f8078e83f33 P 91628bf78d7a4e0b909f85c5b6af76a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:46:58.415993 14666 ts_tablet_manager.cc:1918] T a2f73a46ecba4ae097f51f8078e83f33 P 6f3e51d22a194e7db99512a63ba51e4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:46:58.416116 14666 log.cc:1196] T a2f73a46ecba4ae097f51f8078e83f33 P 6f3e51d22a194e7db99512a63ba51e4f: Deleting WAL directory at /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-1/wal/wals/a2f73a46ecba4ae097f51f8078e83f33
I20260925 04:46:58.416399 14668 ts_tablet_manager.cc:1918] T a2f73a46ecba4ae097f51f8078e83f33 P 91628bf78d7a4e0b909f85c5b6af76a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:46:58.416469 14666 ts_tablet_manager.cc:1939] T a2f73a46ecba4ae097f51f8078e83f33 P 6f3e51d22a194e7db99512a63ba51e4f: Deleting consensus metadata
I20260925 04:46:58.416499 14668 log.cc:1196] T a2f73a46ecba4ae097f51f8078e83f33 P 91628bf78d7a4e0b909f85c5b6af76a4: Deleting WAL directory at /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-2/wal/wals/a2f73a46ecba4ae097f51f8078e83f33
I20260925 04:46:58.416810 14668 ts_tablet_manager.cc:1939] T a2f73a46ecba4ae097f51f8078e83f33 P 91628bf78d7a4e0b909f85c5b6af76a4: Deleting consensus metadata
I20260925 04:46:58.417811 14153 catalog_manager.cc:4841] TS 6f3e51d22a194e7db99512a63ba51e4f (127.0.98.66:44125): tablet a2f73a46ecba4ae097f51f8078e83f33 (table test-table [id=4cdb6641d9c7432d8b5275737dbc9080]) successfully deleted
W20260925 04:46:58.417960 14152 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.65:41417 (error 108)
I20260925 04:46:58.418270 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 14363
I20260925 04:46:58.418357 14151 catalog_manager.cc:4841] TS 91628bf78d7a4e0b909f85c5b6af76a4 (127.0.98.67:43485): tablet a2f73a46ecba4ae097f51f8078e83f33 (table test-table [id=4cdb6641d9c7432d8b5275737dbc9080]) successfully deleted
I20260925 04:46:58.420338 14637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:58.419974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:41417 (local address 127.0.0.1:36774)
0925 04:46:58.420149 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 04:46:58.420233 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.65:41417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 04:46:58.420507 14152 catalog_manager.cc:4584] TS 38d6b9d0fc1649c0bc7c28d0f84bad9e (127.0.98.65:41417): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a2f73a46ecba4ae097f51f8078e83f33: Network error: Client connection negotiation failed: client connection to 127.0.98.65:41417: connect: Connection refused (error 111)
I20260925 04:46:58.424175 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 14497
I20260925 04:46:58.429647 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 14135
I20260925 04:46:58.436058 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:39037
--webserver_interface=127.0.98.126
--webserver_port=42833
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:35919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:39037
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:46:58.545252 14670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:46:58.545512 14670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:46:58.545552 14670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:46:58.549700 14670 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:46:58.549763 14670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:46:58.549783 14670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:46:58.549799 14670 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:46:58.549815 14670 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:46:58.555146 14670 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:35919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:39037
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:39037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=42833
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:46:58.556154 14670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:46:58.557178 14670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:46:58.570283 14677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:46:58.570295 14676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:46:58.570334 14679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:46:58.570549 14670 server_base.cc:1034] running on GCE node
I20260925 04:46:58.571146 14670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:46:58.572238 14670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:46:58.573419 14670 hybrid_clock.cc:648] HybridClock initialized: now 1790311618573393 us; error 46 us; skew 500 ppm
Sep 25 04:46:58 dist-test-slave-kghz krb5kdc[14122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311618, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:46:58.581996 14670 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260925 04:46:58.583074 14670 webserver.cc:460] Webserver started at http://127.0.98.126:42833/ using document root <none> and password file <none>
I20260925 04:46:58.583691 14670 fs_manager.cc:359] Metadata directory not provided
I20260925 04:46:58.583804 14670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:46:58.587990 14670 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260925 04:46:58.590448 14686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:46:58.591809 14670 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 04:46:58.591991 14670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/master-0/wal
uuid: "ae3eeb992c414679aa64a6b0a2d2c0aa"
format_stamp: "Formatted at 2026-09-25 04:46:57 on dist-test-slave-kghz"
server_key: "a89214cbd07d8a9a59f2f8c1fb80a4d8"
server_key_iv: "6fd0c1701b7be61f6b6ffb724fe301ca"
server_key_version: "encryptionkey@0"
I20260925 04:46:58.592631 14670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:46:58.616248 14670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:46:58.619896 14670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:46:58.620126 14670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:46:58.627153 14670 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:39037
I20260925 04:46:58.627473 14738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:39037 every 8 connection(s)
I20260925 04:46:58.628319 14670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
I20260925 04:46:58.632671 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 14670
I20260925 04:46:58.633188 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:41417
--local_ip_for_outbound_sockets=127.0.98.65
--tserver_master_addrs=127.0.98.126:39037
--webserver_port=36377
--webserver_interface=127.0.98.65
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:35919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 04:46:58.633538 14739 sys_catalog.cc:263] Verifying existing consensus state
I20260925 04:46:58.635811 14739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae3eeb992c414679aa64a6b0a2d2c0aa: Bootstrap starting.
I20260925 04:46:58.653255 14739 log.cc:824] T 00000000000000000000000000000000 P ae3eeb992c414679aa64a6b0a2d2c0aa: Log is configured to *not* fsync() on all Append() calls
I20260925 04:46:58.661656 14739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae3eeb992c414679aa64a6b0a2d2c0aa: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 04:46:58.662050 14739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae3eeb992c414679aa64a6b0a2d2c0aa: Bootstrap complete.
I20260925 04:46:58.666338 14739 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae3eeb992c414679aa64a6b0a2d2c0aa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae3eeb992c414679aa64a6b0a2d2c0aa" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39037 } }
I20260925 04:46:58.666754 14739 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae3eeb992c414679aa64a6b0a2d2c0aa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae3eeb992c414679aa64a6b0a2d2c0aa, State: Initialized, Role: FOLLOWER
I20260925 04:46:58.667265 14739 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae3eeb992c414679aa64a6b0a2d2c0aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae3eeb992c414679aa64a6b0a2d2c0aa" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39037 } }
I20260925 04:46:58.667429 14739 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae3eeb992c414679aa64a6b0a2d2c0aa [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:46:58.667528 14739 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae3eeb992c414679aa64a6b0a2d2c0aa [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:46:58.667651 14739 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae3eeb992c414679aa64a6b0a2d2c0aa [term 1 FOLLOWER]: Advancing to term 2
I20260925 04:46:58.669471 14739 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae3eeb992c414679aa64a6b0a2d2c0aa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae3eeb992c414679aa64a6b0a2d2c0aa" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39037 } }
I20260925 04:46:58.669807 14739 leader_election.cc:302] T 00000000000000000000000000000000 P ae3eeb992c414679aa64a6b0a2d2c0aa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae3eeb992c414679aa64a6b0a2d2c0aa; no voters:
I20260925 04:46:58.670137 14739 leader_election.cc:288] T 00000000000000000000000000000000 P ae3eeb992c414679aa64a6b0a2d2c0aa [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 04:46:58.670259 14743 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae3eeb992c414679aa64a6b0a2d2c0aa [term 2 FOLLOWER]: Leader election won for term 2
I20260925 04:46:58.670468 14743 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae3eeb992c414679aa64a6b0a2d2c0aa [term 2 LEADER]: Becoming Leader. State: Replica: ae3eeb992c414679aa64a6b0a2d2c0aa, State: Running, Role: LEADER
I20260925 04:46:58.670787 14743 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae3eeb992c414679aa64a6b0a2d2c0aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae3eeb992c414679aa64a6b0a2d2c0aa" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39037 } }
I20260925 04:46:58.671373 14739 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae3eeb992c414679aa64a6b0a2d2c0aa [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:46:58.672119 14744 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae3eeb992c414679aa64a6b0a2d2c0aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ae3eeb992c414679aa64a6b0a2d2c0aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae3eeb992c414679aa64a6b0a2d2c0aa" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39037 } } }
I20260925 04:46:58.672269 14744 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae3eeb992c414679aa64a6b0a2d2c0aa [sys.catalog]: This master's current role is: LEADER
I20260925 04:46:58.672585 14745 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae3eeb992c414679aa64a6b0a2d2c0aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae3eeb992c414679aa64a6b0a2d2c0aa. Latest consensus state: current_term: 2 leader_uuid: "ae3eeb992c414679aa64a6b0a2d2c0aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae3eeb992c414679aa64a6b0a2d2c0aa" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39037 } } }
I20260925 04:46:58.672729 14745 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae3eeb992c414679aa64a6b0a2d2c0aa [sys.catalog]: This master's current role is: LEADER
I20260925 04:46:58.673228 14752 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:46:58.678362 14752 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:46:58.679167 14752 catalog_manager.cc:1259] Loaded cluster ID: bf5babe13d3e42aba7c2aed4edce0baf
I20260925 04:46:58.679236 14752 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:46:58.680578 14752 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260925 04:46:58.684965 14752 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260925 04:46:58.690485 14752 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:46:58.691291 14752 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ae3eeb992c414679aa64a6b0a2d2c0aa: Loaded TSK: 0
I20260925 04:46:58.692535 14752 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 04:46:58.753170 14741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:46:58.753448 14741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:46:58.753492 14741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:46:58.757678 14741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:46:58.757750 14741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:46:58.757814 14741 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260925 04:46:58.763291 14741 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:35919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:41417
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=36377
--tserver_master_addrs=127.0.98.126:39037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14741
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:46:58.764379 14741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:46:58.765389 14741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:46:58.775254 14767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:46:58.775300 14766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:46:58.775426 14741 server_base.cc:1034] running on GCE node
W20260925 04:46:58.775663 14769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:46:58.776072 14741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:46:58.776798 14741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:46:58.777982 14741 hybrid_clock.cc:648] HybridClock initialized: now 1790311618777961 us; error 36 us; skew 500 ppm
Sep 25 04:46:58 dist-test-slave-kghz krb5kdc[14122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311618, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:46:58.781318 14741 init.cc:377] Logged in from keytab as kudu/127.0.98.65@KRBTEST.COM (short username kudu)
I20260925 04:46:58.782384 14741 webserver.cc:460] Webserver started at http://127.0.98.65:36377/ using document root <none> and password file <none>
I20260925 04:46:58.782990 14741 fs_manager.cc:359] Metadata directory not provided
I20260925 04:46:58.783066 14741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:46:58.787241 14741 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260925 04:46:58.789662 14776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:46:58.791086 14741 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 04:46:58.791235 14741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-0/wal
uuid: "38d6b9d0fc1649c0bc7c28d0f84bad9e"
format_stamp: "Formatted at 2026-09-25 04:46:57 on dist-test-slave-kghz"
server_key: "f0af70068975869c0b1e1a0b0030da27"
server_key_iv: "9926876c9c9692f06e54602eeec8911d"
server_key_version: "encryptionkey@0"
I20260925 04:46:58.791860 14741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:46:58.803615 14741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:46:58.807139 14741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:46:58.807353 14741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:46:58.808045 14741 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:46:58.809170 14783 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 04:46:58.814545 14741 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 04:46:58.814663 14741 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260925 04:46:58.814769 14741 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 04:46:58.816579 14741 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 04:46:58.816663 14741 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260925 04:46:58.816972 14783 tablet_bootstrap.cc:492] T a2f73a46ecba4ae097f51f8078e83f33 P 38d6b9d0fc1649c0bc7c28d0f84bad9e: Bootstrap starting.
I20260925 04:46:58.828193 14741 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:41417
I20260925 04:46:58.828229 14890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:41417 every 8 connection(s)
I20260925 04:46:58.829164 14741 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
I20260925 04:46:58.830091 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 14741
I20260925 04:46:58.830581 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:44125
--local_ip_for_outbound_sockets=127.0.98.66
--tserver_master_addrs=127.0.98.126:39037
--webserver_port=33731
--webserver_interface=127.0.98.66
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:35919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 04:46:58.835759 14783 log.cc:824] T a2f73a46ecba4ae097f51f8078e83f33 P 38d6b9d0fc1649c0bc7c28d0f84bad9e: Log is configured to *not* fsync() on all Append() calls
Sep 25 04:46:58 dist-test-slave-kghz krb5kdc[14122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311618, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:46:58.843310 14783 tablet_bootstrap.cc:492] T a2f73a46ecba4ae097f51f8078e83f33 P 38d6b9d0fc1649c0bc7c28d0f84bad9e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 04:46:58.843761 14783 tablet_bootstrap.cc:492] T a2f73a46ecba4ae097f51f8078e83f33 P 38d6b9d0fc1649c0bc7c28d0f84bad9e: Bootstrap complete.
I20260925 04:46:58.845247 14783 ts_tablet_manager.cc:1397] T a2f73a46ecba4ae097f51f8078e83f33 P 38d6b9d0fc1649c0bc7c28d0f84bad9e: Time spent bootstrapping tablet: real 0.029s user 0.018s sys 0.007s
I20260925 04:46:58.848088 14783 raft_consensus.cc:348] T a2f73a46ecba4ae097f51f8078e83f33 P 38d6b9d0fc1649c0bc7c28d0f84bad9e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f3e51d22a194e7db99512a63ba51e4f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 44125 } } peers { permanent_uuid: "38d6b9d0fc1649c0bc7c28d0f84bad9e" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41417 } } peers { permanent_uuid: "91628bf78d7a4e0b909f85c5b6af76a4" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43485 } }
I20260925 04:46:58.848613 14783 raft_consensus.cc:729] T a2f73a46ecba4ae097f51f8078e83f33 P 38d6b9d0fc1649c0bc7c28d0f84bad9e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38d6b9d0fc1649c0bc7c28d0f84bad9e, State: Initialized, Role: FOLLOWER
I20260925 04:46:58.849155 14783 consensus_queue.cc:260] T a2f73a46ecba4ae097f51f8078e83f33 P 38d6b9d0fc1649c0bc7c28d0f84bad9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f3e51d22a194e7db99512a63ba51e4f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 44125 } } peers { permanent_uuid: "38d6b9d0fc1649c0bc7c28d0f84bad9e" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41417 } } peers { permanent_uuid: "91628bf78d7a4e0b909f85c5b6af76a4" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43485 } }
I20260925 04:46:58.849226 14895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:58.832773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:57531 (local address 127.0.98.126:39037)
0925 04:46:58.833294 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0925 04:46:58.833314 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 04:46:58.833364 (+ 50us) server_negotiation.cc:407] Connection header received
0925 04:46:58.833697 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:46:58.833714 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:46:58.834107 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:46:58.834501 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:46:58.835863 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:58.837630 (+ 1767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:58.839285 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:58.840125 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:58.840327 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:58.843382 (+ 3055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:46:58.843388 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:46:58.843404 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:46:58.843437 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:46:58.845664 (+ 2227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:46:58.846315 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:46:58.846321 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:46:58.846327 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:46:58.846469 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:46:58.846850 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:46:58.846853 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:46:58.846855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:46:58.847301 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:46:58.847433 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 04:46:58.848052 (+ 619us) server_negotiation.cc:299] Negotiation successful
0925 04:46:58.848358 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":116,"threads_started":1}
I20260925 04:46:58.849790 14893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:58.831422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39037 (local address 127.0.98.65:57531)
0925 04:46:58.832054 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0925 04:46:58.832137 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0925 04:46:58.833221 (+ 1084us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:46:58.834558 (+ 1337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:46:58.834568 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:46:58.835018 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:46:58.835654 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:58.835676 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:58.837799 (+ 2123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:58.837802 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:58.839152 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:58.839161 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:58.840280 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:58.840284 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:58.840475 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:58.841202 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:46:58.841221 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:46:58.843168 (+ 1947us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:46:58.845805 (+ 2637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:46:58.845812 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:46:58.845827 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:46:58.846171 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:46:58.846573 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:46:58.846575 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:46:58.846577 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:46:58.846751 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:46:58.847418 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:46:58.847426 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:46:58.847717 (+ 291us) client_negotiation.cc:769] Sending connection context
0925 04:46:58.847912 (+ 195us) client_negotiation.cc:240] Negotiation successful
0925 04:46:58.848247 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":154,"threads_started":1}
I20260925 04:46:58.850077 14783 ts_tablet_manager.cc:1428] T a2f73a46ecba4ae097f51f8078e83f33 P 38d6b9d0fc1649c0bc7c28d0f84bad9e: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260925 04:46:58.852006 14891 heartbeater.cc:347] Connected to a master server at 127.0.98.126:39037
I20260925 04:46:58.852237 14891 heartbeater.cc:464] Registering TS with master...
I20260925 04:46:58.852757 14891 heartbeater.cc:511] Master 127.0.98.126:39037 requested a full tablet report, sending...
I20260925 04:46:58.855137 14703 ts_manager.cc:194] Registered new tserver with Master: 38d6b9d0fc1649c0bc7c28d0f84bad9e (127.0.98.65:41417)
I20260925 04:46:58.858331 14703 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.65@KRBTEST.COM'} at 127.0.98.65:57531
I20260925 04:46:58.860791 14891 heartbeater.cc:503] Master 127.0.98.126:39037 was elected leader, sending a full tablet report...
Sep 25 04:46:58 dist-test-slave-kghz krb5kdc[14122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311618, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for kudu/127.0.98.65@KRBTEST.COM
I20260925 04:46:58.867851 14898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:58.857775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:41417 (local address 127.0.0.1:36778)
0925 04:46:58.858321 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0925 04:46:58.858364 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 04:46:58.858526 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:46:58.859116 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:46:58.859121 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:46:58.859476 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:46:58.859570 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:58.859575 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:58.860489 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:58.860494 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:58.861237 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:58.861249 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:58.861968 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:58.861972 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:58.862086 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:58.862730 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:46:58.862741 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:46:58.864262 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:46:58.866318 (+ 2056us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:46:58.866325 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:46:58.866329 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:46:58.866539 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:46:58.866834 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:46:58.866836 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:46:58.866837 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:46:58.866910 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:46:58.867382 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:46:58.867386 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:46:58.867479 (+ 93us) client_negotiation.cc:769] Sending connection context
0925 04:46:58.867577 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 04:46:58.867726 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":104,"threads_started":1}
I20260925 04:46:58.868011 14899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:58.857851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36778 (local address 127.0.98.65:41417)
0925 04:46:58.858576 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0925 04:46:58.858592 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 04:46:58.858615 (+ 23us) server_negotiation.cc:407] Connection header received
0925 04:46:58.858679 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:46:58.858684 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:46:58.858870 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:46:58.859027 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:46:58.859692 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:58.860375 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:58.861351 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:58.861842 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:58.861955 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:58.864396 (+ 2441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:46:58.864405 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:46:58.864416 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:46:58.864453 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:46:58.866152 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:46:58.866644 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:46:58.866650 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:46:58.866651 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:46:58.866744 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:46:58.866998 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:46:58.867001 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:46:58.867003 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:46:58.867280 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:46:58.867405 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 04:46:58.867696 (+ 291us) server_negotiation.cc:299] Negotiation successful
0925 04:46:58.867861 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":102,"threads_started":1}
I20260925 04:46:58.868974 14825 tablet_service.cc:1514] Processing DeleteTablet for tablet a2f73a46ecba4ae097f51f8078e83f33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:46:58 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:36778
I20260925 04:46:58.869819 14900 tablet_replica.cc:330] stopping tablet replica
I20260925 04:46:58.870131 14900 raft_consensus.cc:2229] T a2f73a46ecba4ae097f51f8078e83f33 P 38d6b9d0fc1649c0bc7c28d0f84bad9e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:46:58.870427 14900 raft_consensus.cc:2258] T a2f73a46ecba4ae097f51f8078e83f33 P 38d6b9d0fc1649c0bc7c28d0f84bad9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:46:58.871534 14900 ts_tablet_manager.cc:1905] T a2f73a46ecba4ae097f51f8078e83f33 P 38d6b9d0fc1649c0bc7c28d0f84bad9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:46:58.876176 14900 ts_tablet_manager.cc:1918] T a2f73a46ecba4ae097f51f8078e83f33 P 38d6b9d0fc1649c0bc7c28d0f84bad9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:46:58.876274 14900 log.cc:1196] T a2f73a46ecba4ae097f51f8078e83f33 P 38d6b9d0fc1649c0bc7c28d0f84bad9e: Deleting WAL directory at /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-0/wal/wals/a2f73a46ecba4ae097f51f8078e83f33
I20260925 04:46:58.876569 14900 ts_tablet_manager.cc:1939] T a2f73a46ecba4ae097f51f8078e83f33 P 38d6b9d0fc1649c0bc7c28d0f84bad9e: Deleting consensus metadata
I20260925 04:46:58.878711 14688 catalog_manager.cc:4841] TS 38d6b9d0fc1649c0bc7c28d0f84bad9e (127.0.98.65:41417): tablet a2f73a46ecba4ae097f51f8078e83f33 (table test-table [id=4cdb6641d9c7432d8b5275737dbc9080]) successfully deleted
W20260925 04:46:58.954267 14894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:46:58.954542 14894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:46:58.954587 14894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:46:58.958637 14894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:46:58.958694 14894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:46:58.958753 14894 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260925 04:46:58.963874 14894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:35919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:44125
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=33731
--tserver_master_addrs=127.0.98.126:39037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:46:58.964941 14894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:46:58.966002 14894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:46:58.975883 14908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:46:58.975922 14905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:46:58.976080 14894 server_base.cc:1034] running on GCE node
W20260925 04:46:58.975886 14906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:46:58.976657 14894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:46:58.977406 14894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:46:58.978657 14894 hybrid_clock.cc:648] HybridClock initialized: now 1790311618978650 us; error 40 us; skew 500 ppm
Sep 25 04:46:58 dist-test-slave-kghz krb5kdc[14122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311618, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:46:58.981532 14894 init.cc:377] Logged in from keytab as kudu/127.0.98.66@KRBTEST.COM (short username kudu)
I20260925 04:46:58.982645 14894 webserver.cc:460] Webserver started at http://127.0.98.66:33731/ using document root <none> and password file <none>
I20260925 04:46:58.983270 14894 fs_manager.cc:359] Metadata directory not provided
I20260925 04:46:58.983384 14894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:46:58.987737 14894 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 04:46:58.990026 14915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:46:58.991359 14894 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:46:58.991513 14894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-1/wal
uuid: "6f3e51d22a194e7db99512a63ba51e4f"
format_stamp: "Formatted at 2026-09-25 04:46:57 on dist-test-slave-kghz"
server_key: "e91bff69d701fad33b0c9d31d1e65f4d"
server_key_iv: "b250dfc98c43168455b1f65fa6b51e7b"
server_key_version: "encryptionkey@0"
I20260925 04:46:58.992100 14894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:46:59.009642 14894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:46:59.012943 14894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:46:59.013177 14894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:46:59.013820 14894 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:46:59.014777 14894 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:46:59.014844 14894 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:46:59.014917 14894 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:46:59.014969 14894 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:46:59.024226 14894 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:44125
I20260925 04:46:59.024245 15028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:44125 every 8 connection(s)
I20260925 04:46:59.025272 14894 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
I20260925 04:46:59.028254 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 14894
I20260925 04:46:59.028744 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:43485
--local_ip_for_outbound_sockets=127.0.98.67
--tserver_master_addrs=127.0.98.126:39037
--webserver_port=39533
--webserver_interface=127.0.98.67
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:35919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 04:46:59 dist-test-slave-kghz krb5kdc[14122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311618, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:46:59.041481 14895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:59.027111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:52903 (local address 127.0.98.126:39037)
0925 04:46:59.027289 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 04:46:59.027293 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:46:59.028211 (+ 918us) server_negotiation.cc:407] Connection header received
0925 04:46:59.030202 (+ 1991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:46:59.030205 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:46:59.030314 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:46:59.030431 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:46:59.031652 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:59.032552 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:59.033827 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:59.034322 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:59.034455 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:59.036936 (+ 2481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:46:59.036943 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:46:59.036948 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:46:59.036984 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:46:59.038909 (+ 1925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:46:59.039654 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:46:59.039676 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:46:59.039679 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:46:59.039844 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:46:59.040276 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:46:59.040282 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:46:59.040286 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:46:59.040536 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:46:59.040644 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 04:46:59.041168 (+ 524us) server_negotiation.cc:299] Negotiation successful
0925 04:46:59.041308 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260925 04:46:59.042408 15031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:59.027498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39037 (local address 127.0.98.66:52903)
0925 04:46:59.027992 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0925 04:46:59.028065 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0925 04:46:59.029916 (+ 1851us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:46:59.030626 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:46:59.030634 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:46:59.031000 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:46:59.031490 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:59.031503 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:59.032690 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:59.032693 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:59.033687 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:59.033697 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:59.034476 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:59.034479 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:59.034636 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:59.035275 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:46:59.035292 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:46:59.036747 (+ 1455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:46:59.039041 (+ 2294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:46:59.039046 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:46:59.039058 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:46:59.039463 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:46:59.039975 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:46:59.039980 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:46:59.039984 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:46:59.040164 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:46:59.040654 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:46:59.040663 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:46:59.040922 (+ 259us) client_negotiation.cc:769] Sending connection context
0925 04:46:59.041126 (+ 204us) client_negotiation.cc:240] Negotiation successful
0925 04:46:59.041470 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":115,"threads_started":1}
I20260925 04:46:59.043727 15029 heartbeater.cc:347] Connected to a master server at 127.0.98.126:39037
I20260925 04:46:59.044013 15029 heartbeater.cc:464] Registering TS with master...
I20260925 04:46:59.044541 15029 heartbeater.cc:511] Master 127.0.98.126:39037 requested a full tablet report, sending...
I20260925 04:46:59.045655 14703 ts_manager.cc:194] Registered new tserver with Master: 6f3e51d22a194e7db99512a63ba51e4f (127.0.98.66:44125)
I20260925 04:46:59.046514 14703 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.66@KRBTEST.COM'} at 127.0.98.66:52903
W20260925 04:46:59.144784 15032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:46:59.145056 15032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:46:59.145099 15032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:46:59.149194 15032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:46:59.149253 15032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:46:59.149344 15032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260925 04:46:59.154577 15032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:35919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:43485
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=39533
--tserver_master_addrs=127.0.98.126:39037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15032
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:46:59.155635 15032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:46:59.156658 15032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:46:59.166224 15038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:46:59.166215 15040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:46:59.166215 15037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:46:59.166419 15032 server_base.cc:1034] running on GCE node
I20260925 04:46:59.166875 15032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:46:59.167579 15032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:46:59.168772 15032 hybrid_clock.cc:648] HybridClock initialized: now 1790311619168756 us; error 26 us; skew 500 ppm
Sep 25 04:46:59 dist-test-slave-kghz krb5kdc[14122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311619, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:46:59.171677 15032 init.cc:377] Logged in from keytab as kudu/127.0.98.67@KRBTEST.COM (short username kudu)
I20260925 04:46:59.172791 15032 webserver.cc:460] Webserver started at http://127.0.98.67:39533/ using document root <none> and password file <none>
I20260925 04:46:59.173434 15032 fs_manager.cc:359] Metadata directory not provided
I20260925 04:46:59.173537 15032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:46:59.177551 15032 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 04:46:59.179781 15047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:46:59.181068 15032 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 04:46:59.181166 15032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-2/wal
uuid: "91628bf78d7a4e0b909f85c5b6af76a4"
format_stamp: "Formatted at 2026-09-25 04:46:58 on dist-test-slave-kghz"
server_key: "91f46b945b0026e354cb60518ec43780"
server_key_iv: "e35f2d4b8d4c010ba7b881e9440673b5"
server_key_version: "encryptionkey@0"
I20260925 04:46:59.181754 15032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:46:59.206148 15032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:46:59.209460 15032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:46:59.209671 15032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:46:59.210319 15032 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:46:59.211182 15032 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:46:59.211230 15032 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:46:59.211269 15032 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:46:59.211323 15032 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:46:59.220408 15032 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:43485
I20260925 04:46:59.220432 15160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:43485 every 8 connection(s)
I20260925 04:46:59.221423 15032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
I20260925 04:46:59.226627 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 15032
Sep 25 04:46:59 dist-test-slave-kghz krb5kdc[14122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311619, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:46:59.235965 15165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:59.227327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50908 (local address 127.0.98.126:39037)
0925 04:46:59.227602 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0925 04:46:59.227607 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:46:59.227727 (+ 120us) server_negotiation.cc:407] Connection header received
0925 04:46:59.227896 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:46:59.227898 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:46:59.227946 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:46:59.228035 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:46:59.228732 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:59.229254 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:59.230105 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:59.230404 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:59.230523 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:59.232152 (+ 1629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:46:59.232157 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:46:59.232159 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:46:59.232185 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:46:59.234300 (+ 2115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:46:59.234726 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:46:59.234730 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:46:59.234732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:46:59.234824 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:46:59.235103 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:46:59.235107 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:46:59.235109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:46:59.235297 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:46:59.235517 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0925 04:46:59.235660 (+ 143us) server_negotiation.cc:299] Negotiation successful
0925 04:46:59.235784 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":102,"threads_started":1}
I20260925 04:46:59.236683 14895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:59.223285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:53881 (local address 127.0.98.126:39037)
0925 04:46:59.223459 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 04:46:59.223464 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:46:59.224343 (+ 879us) server_negotiation.cc:407] Connection header received
0925 04:46:59.225599 (+ 1256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:46:59.225602 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:46:59.225677 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:46:59.225760 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:46:59.227102 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:59.227822 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:59.228996 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:59.229228 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:59.229342 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:59.232352 (+ 3010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:46:59.232355 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:46:59.232357 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:46:59.232394 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:46:59.234579 (+ 2185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:46:59.235075 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:46:59.235080 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:46:59.235082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:46:59.235161 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:46:59.235510 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:46:59.235514 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:46:59.235516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:46:59.235694 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:46:59.235798 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 04:46:59.236423 (+ 625us) server_negotiation.cc:299] Negotiation successful
0925 04:46:59.236550 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 04:46:59.237733 15163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:59.223720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39037 (local address 127.0.98.67:53881)
0925 04:46:59.224182 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0925 04:46:59.224224 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 04:46:59.225382 (+ 1158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:46:59.225927 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:46:59.225934 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:46:59.226394 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:46:59.226947 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:59.226962 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:59.227964 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:59.227968 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:59.228867 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:59.228876 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:59.229385 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:59.229389 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:59.229557 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:59.230171 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:46:59.230188 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:46:59.232197 (+ 2009us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:46:59.234692 (+ 2495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:46:59.234698 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:46:59.234714 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:46:59.234973 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:46:59.235254 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:46:59.235258 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:46:59.235260 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:46:59.235406 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:46:59.235798 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:46:59.235806 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:46:59.236155 (+ 349us) client_negotiation.cc:769] Sending connection context
0925 04:46:59.236367 (+ 212us) client_negotiation.cc:240] Negotiation successful
0925 04:46:59.236707 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":118,"threads_started":1}
I20260925 04:46:59.238938 15161 heartbeater.cc:347] Connected to a master server at 127.0.98.126:39037
I20260925 04:46:59.239192 15161 heartbeater.cc:464] Registering TS with master...
I20260925 04:46:59.239689 15161 heartbeater.cc:511] Master 127.0.98.126:39037 requested a full tablet report, sending...
I20260925 04:46:59.240753 14703 ts_manager.cc:194] Registered new tserver with Master: 91628bf78d7a4e0b909f85c5b6af76a4 (127.0.98.67:43485)
I20260925 04:46:59.241291 14703 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.67@KRBTEST.COM'} at 127.0.98.67:53881
I20260925 04:46:59.249792 393 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 04:46:59.254484 14895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:59.250647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50922 (local address 127.0.98.126:39037)
0925 04:46:59.250795 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 04:46:59.250800 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:46:59.251064 (+ 264us) server_negotiation.cc:407] Connection header received
0925 04:46:59.251255 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:46:59.251257 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:46:59.251366 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:46:59.251457 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:46:59.251679 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:59.252189 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:59.253034 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:59.253257 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:59.253340 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:59.253628 (+ 288us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:46:59.254037 (+ 409us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:46:59.254127 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 04:46:59.254199 (+ 72us) server_negotiation.cc:299] Negotiation successful
0925 04:46:59.254311 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 04:46:59.255291 14703 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:50922:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:46:59.258678 14703 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:46:59.267128 14825 tablet_service.cc:1467] Processing CreateTablet for tablet efd1c3fda547461197fe9c044d10b64b (DEFAULT_TABLE table=test-table [id=083687d3f8124bfe96e1596d1ffb593b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:46:59.267845 14825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efd1c3fda547461197fe9c044d10b64b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:46:59.270429 14783 tablet_bootstrap.cc:492] T efd1c3fda547461197fe9c044d10b64b P 38d6b9d0fc1649c0bc7c28d0f84bad9e: Bootstrap starting.
I20260925 04:46:59.271373 14783 tablet_bootstrap.cc:654] T efd1c3fda547461197fe9c044d10b64b P 38d6b9d0fc1649c0bc7c28d0f84bad9e: Neither blocks nor log segments found. Creating new log.
I20260925 04:46:59.272167 15170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:59.266210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:44125 (local address 127.0.0.1:41558)
0925 04:46:59.266492 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0925 04:46:59.266507 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:46:59.266591 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:46:59.267460 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:46:59.267476 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:46:59.267495 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:46:59.267819 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:59.267825 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:59.268979 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:59.268983 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:59.270575 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:59.270587 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:59.271724 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:59.271729 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:59.271823 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:59.271869 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 04:46:59.271923 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 04:46:59.272017 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"mutex_wait_us":254,"thread_start_us":123,"threads_started":1}
I20260925 04:46:59.272308 15172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:59.266583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56902 (local address 127.0.98.67:43485)
0925 04:46:59.266952 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0925 04:46:59.266965 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 04:46:59.266989 (+ 24us) server_negotiation.cc:407] Connection header received
0925 04:46:59.267067 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:46:59.267072 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:46:59.267286 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:46:59.267638 (+ 352us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:46:59.267940 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:59.268998 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:59.270628 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:59.271578 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:59.271730 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:59.271939 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0925 04:46:59.272055 (+ 116us) server_negotiation.cc:299] Negotiation successful
0925 04:46:59.272148 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":158,"threads_started":1}
I20260925 04:46:59.272320 14898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:59.265926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:43485 (local address 127.0.0.1:56902)
0925 04:46:59.266056 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 04:46:59.266073 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 04:46:59.266157 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:46:59.267458 (+ 1301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:46:59.267462 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:46:59.267478 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:46:59.267679 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:59.267689 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:59.269129 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:59.269132 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:59.270478 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:59.270491 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:59.271746 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:59.271752 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:59.271889 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:59.271921 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 04:46:59.272003 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 04:46:59.272120 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":125}
I20260925 04:46:59.272841 15171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:59.266401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41558 (local address 127.0.98.66:44125)
0925 04:46:59.267002 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0925 04:46:59.267011 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 04:46:59.267030 (+ 19us) server_negotiation.cc:407] Connection header received
0925 04:46:59.267092 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:46:59.267096 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:46:59.267269 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:46:59.267815 (+ 546us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:46:59.268127 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:59.268832 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:59.270699 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:59.271578 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:59.272377 (+ 799us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:59.272555 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0925 04:46:59.272604 (+ 49us) server_negotiation.cc:299] Negotiation successful
0925 04:46:59.272682 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":130,"threads_started":1}
I20260925 04:46:59.273747 14783 tablet_bootstrap.cc:492] T efd1c3fda547461197fe9c044d10b64b P 38d6b9d0fc1649c0bc7c28d0f84bad9e: No bootstrap required, opened a new log
I20260925 04:46:59.273855 14783 ts_tablet_manager.cc:1397] T efd1c3fda547461197fe9c044d10b64b P 38d6b9d0fc1649c0bc7c28d0f84bad9e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 04:46:59.274292 14783 raft_consensus.cc:348] T efd1c3fda547461197fe9c044d10b64b P 38d6b9d0fc1649c0bc7c28d0f84bad9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91628bf78d7a4e0b909f85c5b6af76a4" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43485 } } peers { permanent_uuid: "6f3e51d22a194e7db99512a63ba51e4f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 44125 } } peers { permanent_uuid: "38d6b9d0fc1649c0bc7c28d0f84bad9e" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41417 } }
I20260925 04:46:59.274415 14783 raft_consensus.cc:374] T efd1c3fda547461197fe9c044d10b64b P 38d6b9d0fc1649c0bc7c28d0f84bad9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:46:59.274466 14783 raft_consensus.cc:729] T efd1c3fda547461197fe9c044d10b64b P 38d6b9d0fc1649c0bc7c28d0f84bad9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38d6b9d0fc1649c0bc7c28d0f84bad9e, State: Initialized, Role: FOLLOWER
I20260925 04:46:59.274793 14783 consensus_queue.cc:260] T efd1c3fda547461197fe9c044d10b64b P 38d6b9d0fc1649c0bc7c28d0f84bad9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91628bf78d7a4e0b909f85c5b6af76a4" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43485 } } peers { permanent_uuid: "6f3e51d22a194e7db99512a63ba51e4f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 44125 } } peers { permanent_uuid: "38d6b9d0fc1649c0bc7c28d0f84bad9e" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41417 } }
I20260925 04:46:59.275641 14783 ts_tablet_manager.cc:1428] T efd1c3fda547461197fe9c044d10b64b P 38d6b9d0fc1649c0bc7c28d0f84bad9e: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260925 04:46:59.275635 15095 tablet_service.cc:1467] Processing CreateTablet for tablet efd1c3fda547461197fe9c044d10b64b (DEFAULT_TABLE table=test-table [id=083687d3f8124bfe96e1596d1ffb593b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:46:59.275635 14963 tablet_service.cc:1467] Processing CreateTablet for tablet efd1c3fda547461197fe9c044d10b64b (DEFAULT_TABLE table=test-table [id=083687d3f8124bfe96e1596d1ffb593b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:46:59.276780 15095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efd1c3fda547461197fe9c044d10b64b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:46:59.276777 14963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efd1c3fda547461197fe9c044d10b64b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:46:59.285248 15175 tablet_bootstrap.cc:492] T efd1c3fda547461197fe9c044d10b64b P 6f3e51d22a194e7db99512a63ba51e4f: Bootstrap starting.
I20260925 04:46:59.285248 15174 tablet_bootstrap.cc:492] T efd1c3fda547461197fe9c044d10b64b P 91628bf78d7a4e0b909f85c5b6af76a4: Bootstrap starting.
I20260925 04:46:59.288008 15174 tablet_bootstrap.cc:654] T efd1c3fda547461197fe9c044d10b64b P 91628bf78d7a4e0b909f85c5b6af76a4: Neither blocks nor log segments found. Creating new log.
I20260925 04:46:59.288477 15175 tablet_bootstrap.cc:654] T efd1c3fda547461197fe9c044d10b64b P 6f3e51d22a194e7db99512a63ba51e4f: Neither blocks nor log segments found. Creating new log.
I20260925 04:46:59.289383 15175 log.cc:824] T efd1c3fda547461197fe9c044d10b64b P 6f3e51d22a194e7db99512a63ba51e4f: Log is configured to *not* fsync() on all Append() calls
I20260925 04:46:59.289382 15174 log.cc:824] T efd1c3fda547461197fe9c044d10b64b P 91628bf78d7a4e0b909f85c5b6af76a4: Log is configured to *not* fsync() on all Append() calls
I20260925 04:46:59.291641 15175 tablet_bootstrap.cc:492] T efd1c3fda547461197fe9c044d10b64b P 6f3e51d22a194e7db99512a63ba51e4f: No bootstrap required, opened a new log
I20260925 04:46:59.291641 15174 tablet_bootstrap.cc:492] T efd1c3fda547461197fe9c044d10b64b P 91628bf78d7a4e0b909f85c5b6af76a4: No bootstrap required, opened a new log
I20260925 04:46:59.291852 15175 ts_tablet_manager.cc:1397] T efd1c3fda547461197fe9c044d10b64b P 6f3e51d22a194e7db99512a63ba51e4f: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 04:46:59.291852 15174 ts_tablet_manager.cc:1397] T efd1c3fda547461197fe9c044d10b64b P 91628bf78d7a4e0b909f85c5b6af76a4: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260925 04:46:59.295979 15175 raft_consensus.cc:348] T efd1c3fda547461197fe9c044d10b64b P 6f3e51d22a194e7db99512a63ba51e4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91628bf78d7a4e0b909f85c5b6af76a4" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43485 } } peers { permanent_uuid: "6f3e51d22a194e7db99512a63ba51e4f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 44125 } } peers { permanent_uuid: "38d6b9d0fc1649c0bc7c28d0f84bad9e" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41417 } }
I20260925 04:46:59.295979 15174 raft_consensus.cc:348] T efd1c3fda547461197fe9c044d10b64b P 91628bf78d7a4e0b909f85c5b6af76a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91628bf78d7a4e0b909f85c5b6af76a4" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43485 } } peers { permanent_uuid: "6f3e51d22a194e7db99512a63ba51e4f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 44125 } } peers { permanent_uuid: "38d6b9d0fc1649c0bc7c28d0f84bad9e" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41417 } }
I20260925 04:46:59.296288 15175 raft_consensus.cc:374] T efd1c3fda547461197fe9c044d10b64b P 6f3e51d22a194e7db99512a63ba51e4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:46:59.296288 15174 raft_consensus.cc:374] T efd1c3fda547461197fe9c044d10b64b P 91628bf78d7a4e0b909f85c5b6af76a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:46:59.296346 15175 raft_consensus.cc:729] T efd1c3fda547461197fe9c044d10b64b P 6f3e51d22a194e7db99512a63ba51e4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f3e51d22a194e7db99512a63ba51e4f, State: Initialized, Role: FOLLOWER
I20260925 04:46:59.296346 15174 raft_consensus.cc:729] T efd1c3fda547461197fe9c044d10b64b P 91628bf78d7a4e0b909f85c5b6af76a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91628bf78d7a4e0b909f85c5b6af76a4, State: Initialized, Role: FOLLOWER
I20260925 04:46:59.296826 15175 consensus_queue.cc:260] T efd1c3fda547461197fe9c044d10b64b P 6f3e51d22a194e7db99512a63ba51e4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91628bf78d7a4e0b909f85c5b6af76a4" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43485 } } peers { permanent_uuid: "6f3e51d22a194e7db99512a63ba51e4f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 44125 } } peers { permanent_uuid: "38d6b9d0fc1649c0bc7c28d0f84bad9e" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41417 } }
I20260925 04:46:59.296826 15174 consensus_queue.cc:260] T efd1c3fda547461197fe9c044d10b64b P 91628bf78d7a4e0b909f85c5b6af76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91628bf78d7a4e0b909f85c5b6af76a4" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43485 } } peers { permanent_uuid: "6f3e51d22a194e7db99512a63ba51e4f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 44125 } } peers { permanent_uuid: "38d6b9d0fc1649c0bc7c28d0f84bad9e" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41417 } }
I20260925 04:46:59.297530 15029 heartbeater.cc:503] Master 127.0.98.126:39037 was elected leader, sending a full tablet report...
I20260925 04:46:59.297777 15161 heartbeater.cc:503] Master 127.0.98.126:39037 was elected leader, sending a full tablet report...
I20260925 04:46:59.297907 15175 ts_tablet_manager.cc:1428] T efd1c3fda547461197fe9c044d10b64b P 6f3e51d22a194e7db99512a63ba51e4f: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 04:46:59.297907 15174 ts_tablet_manager.cc:1428] T efd1c3fda547461197fe9c044d10b64b P 91628bf78d7a4e0b909f85c5b6af76a4: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260925 04:46:59.330485 14892 tablet.cc:2369] T efd1c3fda547461197fe9c044d10b64b P 38d6b9d0fc1649c0bc7c28d0f84bad9e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:46:59.378088 15178 raft_consensus.cc:482] T efd1c3fda547461197fe9c044d10b64b P 6f3e51d22a194e7db99512a63ba51e4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:46:59.378357 15178 raft_consensus.cc:504] T efd1c3fda547461197fe9c044d10b64b P 6f3e51d22a194e7db99512a63ba51e4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91628bf78d7a4e0b909f85c5b6af76a4" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43485 } } peers { permanent_uuid: "6f3e51d22a194e7db99512a63ba51e4f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 44125 } } peers { permanent_uuid: "38d6b9d0fc1649c0bc7c28d0f84bad9e" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41417 } }
I20260925 04:46:59.379622 15178 leader_election.cc:288] T efd1c3fda547461197fe9c044d10b64b P 6f3e51d22a194e7db99512a63ba51e4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91628bf78d7a4e0b909f85c5b6af76a4 (127.0.98.67:43485), 38d6b9d0fc1649c0bc7c28d0f84bad9e (127.0.98.65:41417)
I20260925 04:46:59.383517 15172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:59.380018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:36773 (local address 127.0.98.67:43485)
0925 04:46:59.380165 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 04:46:59.380171 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:46:59.380441 (+ 270us) server_negotiation.cc:407] Connection header received
0925 04:46:59.380601 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:46:59.380604 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:46:59.380648 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:46:59.380727 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:46:59.380983 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:59.381476 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:59.382557 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:59.382961 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:59.383076 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:59.383147 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 04:46:59.383322 (+ 175us) server_negotiation.cc:299] Negotiation successful
0925 04:46:59.383389 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 04:46:59.383535 15181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:59.379903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:43485 (local address 127.0.98.66:36773)
0925 04:46:59.380344 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0925 04:46:59.380356 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 04:46:59.380483 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:46:59.380748 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:46:59.380751 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:46:59.380766 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:46:59.380881 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:59.380886 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:59.381586 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:59.381591 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:59.382440 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:59.382449 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:59.383092 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:59.383096 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:59.383196 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:59.383221 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 04:46:59.383306 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 04:46:59.383379 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"mutex_wait_us":120,"thread_start_us":118,"threads_started":1}
I20260925 04:46:59.383673 15180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:59.380044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:35001 (local address 127.0.98.65:41417)
0925 04:46:59.380296 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0925 04:46:59.380307 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 04:46:59.380319 (+ 12us) server_negotiation.cc:407] Connection header received
0925 04:46:59.380364 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:46:59.380367 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:46:59.380415 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:46:59.380532 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:46:59.380815 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:59.381347 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:59.382562 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:59.383001 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:59.383208 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:59.383394 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0925 04:46:59.383448 (+ 54us) server_negotiation.cc:299] Negotiation successful
0925 04:46:59.383517 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":74,"threads_started":1}
I20260925 04:46:59.384050 15031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:59.379903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:41417 (local address 127.0.98.66:35001)
0925 04:46:59.380044 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 04:46:59.380061 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 04:46:59.380180 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:46:59.380556 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:46:59.380559 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:46:59.380577 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:46:59.380701 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:59.380706 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:59.381464 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:59.381468 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:59.382401 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:46:59.382410 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:46:59.383205 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:46:59.383209 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:46:59.383319 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:59.383339 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 04:46:59.383403 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 04:46:59.383490 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":53}
I20260925 04:46:59.384193 15115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efd1c3fda547461197fe9c044d10b64b" candidate_uuid: "6f3e51d22a194e7db99512a63ba51e4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91628bf78d7a4e0b909f85c5b6af76a4" is_pre_election: true
I20260925 04:46:59.384564 15115 raft_consensus.cc:2454] T efd1c3fda547461197fe9c044d10b64b P 91628bf78d7a4e0b909f85c5b6af76a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f3e51d22a194e7db99512a63ba51e4f in term 0.
I20260925 04:46:59.384642 14845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efd1c3fda547461197fe9c044d10b64b" candidate_uuid: "6f3e51d22a194e7db99512a63ba51e4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38d6b9d0fc1649c0bc7c28d0f84bad9e" is_pre_election: true
I20260925 04:46:59.384960 14845 raft_consensus.cc:2454] T efd1c3fda547461197fe9c044d10b64b P 38d6b9d0fc1649c0bc7c28d0f84bad9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f3e51d22a194e7db99512a63ba51e4f in term 0.
I20260925 04:46:59.385105 14916 leader_election.cc:302] T efd1c3fda547461197fe9c044d10b64b P 6f3e51d22a194e7db99512a63ba51e4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f3e51d22a194e7db99512a63ba51e4f, 91628bf78d7a4e0b909f85c5b6af76a4; no voters:
I20260925 04:46:59.385418 15178 raft_consensus.cc:2781] T efd1c3fda547461197fe9c044d10b64b P 6f3e51d22a194e7db99512a63ba51e4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:46:59.385494 15178 raft_consensus.cc:482] T efd1c3fda547461197fe9c044d10b64b P 6f3e51d22a194e7db99512a63ba51e4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:46:59.385524 15178 raft_consensus.cc:3037] T efd1c3fda547461197fe9c044d10b64b P 6f3e51d22a194e7db99512a63ba51e4f [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:46:59.386670 15178 raft_consensus.cc:504] T efd1c3fda547461197fe9c044d10b64b P 6f3e51d22a194e7db99512a63ba51e4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91628bf78d7a4e0b909f85c5b6af76a4" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43485 } } peers { permanent_uuid: "6f3e51d22a194e7db99512a63ba51e4f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 44125 } } peers { permanent_uuid: "38d6b9d0fc1649c0bc7c28d0f84bad9e" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41417 } }
I20260925 04:46:59.387036 15178 leader_election.cc:288] T efd1c3fda547461197fe9c044d10b64b P 6f3e51d22a194e7db99512a63ba51e4f [CANDIDATE]: Term 1 election: Requested vote from peers 91628bf78d7a4e0b909f85c5b6af76a4 (127.0.98.67:43485), 38d6b9d0fc1649c0bc7c28d0f84bad9e (127.0.98.65:41417)
I20260925 04:46:59.387372 14845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efd1c3fda547461197fe9c044d10b64b" candidate_uuid: "6f3e51d22a194e7db99512a63ba51e4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38d6b9d0fc1649c0bc7c28d0f84bad9e"
I20260925 04:46:59.387411 15115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efd1c3fda547461197fe9c044d10b64b" candidate_uuid: "6f3e51d22a194e7db99512a63ba51e4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91628bf78d7a4e0b909f85c5b6af76a4"
I20260925 04:46:59.387487 14845 raft_consensus.cc:3037] T efd1c3fda547461197fe9c044d10b64b P 38d6b9d0fc1649c0bc7c28d0f84bad9e [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:46:59.387566 15115 raft_consensus.cc:3037] T efd1c3fda547461197fe9c044d10b64b P 91628bf78d7a4e0b909f85c5b6af76a4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:46:59.388535 14845 raft_consensus.cc:2454] T efd1c3fda547461197fe9c044d10b64b P 38d6b9d0fc1649c0bc7c28d0f84bad9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f3e51d22a194e7db99512a63ba51e4f in term 1.
I20260925 04:46:59.388700 15115 raft_consensus.cc:2454] T efd1c3fda547461197fe9c044d10b64b P 91628bf78d7a4e0b909f85c5b6af76a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f3e51d22a194e7db99512a63ba51e4f in term 1.
I20260925 04:46:59.388836 14917 leader_election.cc:302] T efd1c3fda547461197fe9c044d10b64b P 6f3e51d22a194e7db99512a63ba51e4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38d6b9d0fc1649c0bc7c28d0f84bad9e, 6f3e51d22a194e7db99512a63ba51e4f; no voters:
I20260925 04:46:59.389012 15178 raft_consensus.cc:2781] T efd1c3fda547461197fe9c044d10b64b P 6f3e51d22a194e7db99512a63ba51e4f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:46:59.389248 15178 raft_consensus.cc:686] T efd1c3fda547461197fe9c044d10b64b P 6f3e51d22a194e7db99512a63ba51e4f [term 1 LEADER]: Becoming Leader. State: Replica: 6f3e51d22a194e7db99512a63ba51e4f, State: Running, Role: LEADER
I20260925 04:46:59.389577 15178 consensus_queue.cc:237] T efd1c3fda547461197fe9c044d10b64b P 6f3e51d22a194e7db99512a63ba51e4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91628bf78d7a4e0b909f85c5b6af76a4" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43485 } } peers { permanent_uuid: "6f3e51d22a194e7db99512a63ba51e4f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 44125 } } peers { permanent_uuid: "38d6b9d0fc1649c0bc7c28d0f84bad9e" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41417 } }
I20260925 04:46:59.392768 14701 catalog_manager.cc:5495] T efd1c3fda547461197fe9c044d10b64b P 6f3e51d22a194e7db99512a63ba51e4f reported cstate change: term changed from 0 to 1, leader changed from <none> to 6f3e51d22a194e7db99512a63ba51e4f (127.0.98.66). New cstate: current_term: 1 leader_uuid: "6f3e51d22a194e7db99512a63ba51e4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91628bf78d7a4e0b909f85c5b6af76a4" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f3e51d22a194e7db99512a63ba51e4f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 44125 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38d6b9d0fc1649c0bc7c28d0f84bad9e" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41417 } health_report { overall_health: UNKNOWN } } }
I20260925 04:46:59.430182 15171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:59.426783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37254 (local address 127.0.98.66:44125)
0925 04:46:59.426920 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 04:46:59.426924 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:46:59.426937 (+ 13us) server_negotiation.cc:407] Connection header received
0925 04:46:59.427062 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:46:59.427065 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:46:59.427123 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:46:59.427214 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:46:59.427472 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:59.427978 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:59.428836 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:46:59.429040 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:46:59.429114 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:46:59.429390 (+ 276us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:46:59.429801 (+ 411us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:46:59.429886 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 04:46:59.430004 (+ 118us) server_negotiation.cc:299] Negotiation successful
0925 04:46:59.430053 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 04:46:59.435916 14845 raft_consensus.cc:1261] T efd1c3fda547461197fe9c044d10b64b P 38d6b9d0fc1649c0bc7c28d0f84bad9e [term 1 FOLLOWER]: Refusing update from remote peer 6f3e51d22a194e7db99512a63ba51e4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:46:59.435920 15115 raft_consensus.cc:1261] T efd1c3fda547461197fe9c044d10b64b P 91628bf78d7a4e0b909f85c5b6af76a4 [term 1 FOLLOWER]: Refusing update from remote peer 6f3e51d22a194e7db99512a63ba51e4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:46:59.436618 15178 consensus_queue.cc:1035] T efd1c3fda547461197fe9c044d10b64b P 6f3e51d22a194e7db99512a63ba51e4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38d6b9d0fc1649c0bc7c28d0f84bad9e" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:46:59.436834 15182 consensus_queue.cc:1035] T efd1c3fda547461197fe9c044d10b64b P 6f3e51d22a194e7db99512a63ba51e4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "91628bf78d7a4e0b909f85c5b6af76a4" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:46:59.444443 15186 mvcc.cc:205] Tried to move back new op lower bound from 7333116393202208768 to 7333116393022304256. Current Snapshot: MvccSnapshot[applied={T|T < 7333116393202208768 or (T in {7333116393202208768})}]
I20260925 04:46:59.445286 15187 mvcc.cc:205] Tried to move back new op lower bound from 7333116393202208768 to 7333116393022304256. Current Snapshot: MvccSnapshot[applied={T|T < 7333116393202208768}]
I20260925 04:46:59.452193 14702 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:50922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:46:59.452344 14702 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:50922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:46:59.454854 14702 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ae3eeb992c414679aa64a6b0a2d2c0aa: Sending DeleteTablet for 3 replicas of tablet efd1c3fda547461197fe9c044d10b64b
I20260925 04:46:59.455576 15095 tablet_service.cc:1514] Processing DeleteTablet for tablet efd1c3fda547461197fe9c044d10b64b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:46:59 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:56902
I20260925 04:46:59.455641 14963 tablet_service.cc:1514] Processing DeleteTablet for tablet efd1c3fda547461197fe9c044d10b64b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:46:59 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:41558
I20260925 04:46:59.455720 14825 tablet_service.cc:1514] Processing DeleteTablet for tablet efd1c3fda547461197fe9c044d10b64b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:46:59 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:36778
I20260925 04:46:59.456022 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 14741
I20260925 04:46:59.456105 15196 tablet_replica.cc:330] T efd1c3fda547461197fe9c044d10b64b P 6f3e51d22a194e7db99512a63ba51e4f: stopping tablet replica
I20260925 04:46:59.456419 15196 raft_consensus.cc:2229] T efd1c3fda547461197fe9c044d10b64b P 6f3e51d22a194e7db99512a63ba51e4f [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:46:59.456785 15196 raft_consensus.cc:2258] T efd1c3fda547461197fe9c044d10b64b P 6f3e51d22a194e7db99512a63ba51e4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:46:59.458366 15196 ts_tablet_manager.cc:1905] T efd1c3fda547461197fe9c044d10b64b P 6f3e51d22a194e7db99512a63ba51e4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:46:59.459313 15194 tablet_replica.cc:330] T efd1c3fda547461197fe9c044d10b64b P 91628bf78d7a4e0b909f85c5b6af76a4: stopping tablet replica
I20260925 04:46:59.459619 15194 raft_consensus.cc:2229] T efd1c3fda547461197fe9c044d10b64b P 91628bf78d7a4e0b909f85c5b6af76a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:46:59.459950 15194 raft_consensus.cc:2258] T efd1c3fda547461197fe9c044d10b64b P 91628bf78d7a4e0b909f85c5b6af76a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:46:59.461496 15194 ts_tablet_manager.cc:1905] T efd1c3fda547461197fe9c044d10b64b P 91628bf78d7a4e0b909f85c5b6af76a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:46:59.461584 15196 ts_tablet_manager.cc:1918] T efd1c3fda547461197fe9c044d10b64b P 6f3e51d22a194e7db99512a63ba51e4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:46:59.461699 15196 log.cc:1196] T efd1c3fda547461197fe9c044d10b64b P 6f3e51d22a194e7db99512a63ba51e4f: Deleting WAL directory at /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-1/wal/wals/efd1c3fda547461197fe9c044d10b64b
I20260925 04:46:59.462031 15196 ts_tablet_manager.cc:1939] T efd1c3fda547461197fe9c044d10b64b P 6f3e51d22a194e7db99512a63ba51e4f: Deleting consensus metadata
I20260925 04:46:59.463313 14689 catalog_manager.cc:4841] TS 6f3e51d22a194e7db99512a63ba51e4f (127.0.98.66:44125): tablet efd1c3fda547461197fe9c044d10b64b (table test-table [id=083687d3f8124bfe96e1596d1ffb593b]) successfully deleted
I20260925 04:46:59.464504 15194 ts_tablet_manager.cc:1918] T efd1c3fda547461197fe9c044d10b64b P 91628bf78d7a4e0b909f85c5b6af76a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:46:59.464700 15194 log.cc:1196] T efd1c3fda547461197fe9c044d10b64b P 91628bf78d7a4e0b909f85c5b6af76a4: Deleting WAL directory at /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-2/wal/wals/efd1c3fda547461197fe9c044d10b64b
I20260925 04:46:59.465049 15194 ts_tablet_manager.cc:1939] T efd1c3fda547461197fe9c044d10b64b P 91628bf78d7a4e0b909f85c5b6af76a4: Deleting consensus metadata
W20260925 04:46:59.465736 14688 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.65:41417 (error 108)
I20260925 04:46:59.466037 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 14894
I20260925 04:46:59.466212 14687 catalog_manager.cc:4841] TS 91628bf78d7a4e0b909f85c5b6af76a4 (127.0.98.67:43485): tablet efd1c3fda547461197fe9c044d10b64b (table test-table [id=083687d3f8124bfe96e1596d1ffb593b]) successfully deleted
I20260925 04:46:59.468006 14898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:59.467731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:41417 (local address 127.0.0.1:46840)
0925 04:46:59.467866 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 04:46:59.467937 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.65:41417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260925 04:46:59.468250 14688 catalog_manager.cc:4584] TS 38d6b9d0fc1649c0bc7c28d0f84bad9e (127.0.98.65:41417): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet efd1c3fda547461197fe9c044d10b64b: Network error: Client connection negotiation failed: client connection to 127.0.98.65:41417: connect: Connection refused (error 111)
I20260925 04:46:59.471699 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 15032
I20260925 04:46:59.477336 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 14670
I20260925 04:46:59.483770 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:39037
--webserver_interface=127.0.98.126
--webserver_port=42833
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:35919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:39037
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:46:59.595285 15198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:46:59.595607 15198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:46:59.595705 15198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:46:59.599858 15198 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:46:59.599941 15198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:46:59.599977 15198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:46:59.600018 15198 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:46:59.600055 15198 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:46:59.605432 15198 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:35919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:39037
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:39037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=42833
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15198
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:46:59.606542 15198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:46:59.607569 15198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:46:59.616171 15206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:46:59.616182 15198 server_base.cc:1034] running on GCE node
W20260925 04:46:59.616207 15204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:46:59.616207 15203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:46:59.617075 15198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:46:59.618184 15198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:46:59.619398 15198 hybrid_clock.cc:648] HybridClock initialized: now 1790311619619382 us; error 31 us; skew 500 ppm
Sep 25 04:46:59 dist-test-slave-kghz krb5kdc[14122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311619, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:46:59.622121 15198 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260925 04:46:59.623180 15198 webserver.cc:460] Webserver started at http://127.0.98.126:42833/ using document root <none> and password file <none>
I20260925 04:46:59.623754 15198 fs_manager.cc:359] Metadata directory not provided
I20260925 04:46:59.623845 15198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:46:59.627678 15198 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260925 04:46:59.630184 15213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:46:59.631601 15198 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:46:59.631759 15198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/master-0/wal
uuid: "ae3eeb992c414679aa64a6b0a2d2c0aa"
format_stamp: "Formatted at 2026-09-25 04:46:57 on dist-test-slave-kghz"
server_key: "a89214cbd07d8a9a59f2f8c1fb80a4d8"
server_key_iv: "6fd0c1701b7be61f6b6ffb724fe301ca"
server_key_version: "encryptionkey@0"
I20260925 04:46:59.632356 15198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:46:59.666435 15198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:46:59.669411 15198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:46:59.669648 15198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:46:59.677012 15198 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:39037
I20260925 04:46:59.677312 15265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:39037 every 8 connection(s)
I20260925 04:46:59.678151 15198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
I20260925 04:46:59.680320 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 15198
I20260925 04:46:59.680757 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:41417
--local_ip_for_outbound_sockets=127.0.98.65
--tserver_master_addrs=127.0.98.126:39037
--webserver_port=36377
--webserver_interface=127.0.98.65
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:35919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 04:46:59.682683 15266 sys_catalog.cc:263] Verifying existing consensus state
I20260925 04:46:59.684361 15266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae3eeb992c414679aa64a6b0a2d2c0aa: Bootstrap starting.
I20260925 04:46:59.701473 15266 log.cc:824] T 00000000000000000000000000000000 P ae3eeb992c414679aa64a6b0a2d2c0aa: Log is configured to *not* fsync() on all Append() calls
I20260925 04:46:59.712878 15266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae3eeb992c414679aa64a6b0a2d2c0aa: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 04:46:59.713316 15266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae3eeb992c414679aa64a6b0a2d2c0aa: Bootstrap complete.
I20260925 04:46:59.717597 15266 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae3eeb992c414679aa64a6b0a2d2c0aa [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae3eeb992c414679aa64a6b0a2d2c0aa" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39037 } }
I20260925 04:46:59.718014 15266 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae3eeb992c414679aa64a6b0a2d2c0aa [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae3eeb992c414679aa64a6b0a2d2c0aa, State: Initialized, Role: FOLLOWER
I20260925 04:46:59.718570 15266 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae3eeb992c414679aa64a6b0a2d2c0aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae3eeb992c414679aa64a6b0a2d2c0aa" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39037 } }
I20260925 04:46:59.718720 15266 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae3eeb992c414679aa64a6b0a2d2c0aa [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:46:59.718812 15266 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae3eeb992c414679aa64a6b0a2d2c0aa [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:46:59.718954 15266 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae3eeb992c414679aa64a6b0a2d2c0aa [term 2 FOLLOWER]: Advancing to term 3
I20260925 04:46:59.720960 15266 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae3eeb992c414679aa64a6b0a2d2c0aa [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae3eeb992c414679aa64a6b0a2d2c0aa" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39037 } }
I20260925 04:46:59.721308 15266 leader_election.cc:302] T 00000000000000000000000000000000 P ae3eeb992c414679aa64a6b0a2d2c0aa [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae3eeb992c414679aa64a6b0a2d2c0aa; no voters:
I20260925 04:46:59.721683 15266 leader_election.cc:288] T 00000000000000000000000000000000 P ae3eeb992c414679aa64a6b0a2d2c0aa [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 04:46:59.721761 15270 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae3eeb992c414679aa64a6b0a2d2c0aa [term 3 FOLLOWER]: Leader election won for term 3
I20260925 04:46:59.721952 15270 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae3eeb992c414679aa64a6b0a2d2c0aa [term 3 LEADER]: Becoming Leader. State: Replica: ae3eeb992c414679aa64a6b0a2d2c0aa, State: Running, Role: LEADER
I20260925 04:46:59.722273 15270 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae3eeb992c414679aa64a6b0a2d2c0aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae3eeb992c414679aa64a6b0a2d2c0aa" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39037 } }
I20260925 04:46:59.722952 15266 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae3eeb992c414679aa64a6b0a2d2c0aa [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:46:59.723237 15270 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae3eeb992c414679aa64a6b0a2d2c0aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae3eeb992c414679aa64a6b0a2d2c0aa. Latest consensus state: current_term: 3 leader_uuid: "ae3eeb992c414679aa64a6b0a2d2c0aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae3eeb992c414679aa64a6b0a2d2c0aa" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39037 } } }
I20260925 04:46:59.723351 15270 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae3eeb992c414679aa64a6b0a2d2c0aa [sys.catalog]: This master's current role is: LEADER
I20260925 04:46:59.723709 15274 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:46:59.723789 15271 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae3eeb992c414679aa64a6b0a2d2c0aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "ae3eeb992c414679aa64a6b0a2d2c0aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae3eeb992c414679aa64a6b0a2d2c0aa" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39037 } } }
I20260925 04:46:59.723978 15271 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae3eeb992c414679aa64a6b0a2d2c0aa [sys.catalog]: This master's current role is: LEADER
I20260925 04:46:59.728993 15274 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:46:59.729864 15274 catalog_manager.cc:1259] Loaded cluster ID: bf5babe13d3e42aba7c2aed4edce0baf
I20260925 04:46:59.729928 15274 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260925 04:46:59.730950 15274 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f6ffc1729ad google::LogMessage::Fail() at ??:0
@ 0x7f6ffc17588a google::LogMessage::SendToLog() at ??:0
@ 0x7f6ffc1724de google::LogMessage::Flush() at ??:0
@ 0x7f6ffc173a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f7006ab1fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f7006ab2d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f7006aaf579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f7006af3640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f6ffe3c43d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f6ffe3c4c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f6ffe3c63f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f6ffe3b5512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f6fff9f96db start_thread at ??:0
@ 0x7f6ffac7061f clone at ??:0
W20260925 04:46:59.821718 15268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:46:59.822036 15268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:46:59.822111 15268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:46:59.826339 15268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:46:59.826427 15268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:46:59.826545 15268 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260925 04:46:59.832129 15268 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:35919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:41417
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=36377
--tserver_master_addrs=127.0.98.126:39037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15268
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:46:59.833280 15268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:46:59.834403 15268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:46:59.843987 15307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:46:59.844081 15310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:46:59.844026 15308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:46:59.844394 15268 server_base.cc:1034] running on GCE node
I20260925 04:46:59.844844 15268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:46:59.845674 15268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:46:59.846840 15268 hybrid_clock.cc:648] HybridClock initialized: now 1790311619846806 us; error 49 us; skew 500 ppm
Sep 25 04:46:59 dist-test-slave-kghz krb5kdc[14122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311619, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:46:59.850003 15268 init.cc:377] Logged in from keytab as kudu/127.0.98.65@KRBTEST.COM (short username kudu)
I20260925 04:46:59.850984 15268 webserver.cc:460] Webserver started at http://127.0.98.65:36377/ using document root <none> and password file <none>
I20260925 04:46:59.851576 15268 fs_manager.cc:359] Metadata directory not provided
I20260925 04:46:59.851672 15268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:46:59.855746 15268 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260925 04:46:59.858084 15317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:46:59.859510 15268 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 04:46:59.859661 15268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-0/wal
uuid: "38d6b9d0fc1649c0bc7c28d0f84bad9e"
format_stamp: "Formatted at 2026-09-25 04:46:57 on dist-test-slave-kghz"
server_key: "f0af70068975869c0b1e1a0b0030da27"
server_key_iv: "9926876c9c9692f06e54602eeec8911d"
server_key_version: "encryptionkey@0"
I20260925 04:46:59.860226 15268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:46:59.892450 15268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:46:59.896718 15268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:46:59.896938 15268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:46:59.897775 15268 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:46:59.899444 15324 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 04:46:59.903627 15268 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 04:46:59.903712 15268 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260925 04:46:59.903781 15268 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 04:46:59.906087 15268 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 04:46:59.906152 15268 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260925 04:46:59.906528 15324 tablet_bootstrap.cc:492] T efd1c3fda547461197fe9c044d10b64b P 38d6b9d0fc1649c0bc7c28d0f84bad9e: Bootstrap starting.
I20260925 04:46:59.917606 15268 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:41417
I20260925 04:46:59.917665 15431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:41417 every 8 connection(s)
I20260925 04:46:59.918895 15268 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
I20260925 04:46:59.927752 15324 log.cc:824] T efd1c3fda547461197fe9c044d10b64b P 38d6b9d0fc1649c0bc7c28d0f84bad9e: Log is configured to *not* fsync() on all Append() calls
I20260925 04:46:59.928303 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 15268
I20260925 04:46:59.928776 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:44125
--local_ip_for_outbound_sockets=127.0.98.66
--tserver_master_addrs=127.0.98.126:39037
--webserver_port=33731
--webserver_interface=127.0.98.66
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:35919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 04:46:59.936112 15324 tablet_bootstrap.cc:492] T efd1c3fda547461197fe9c044d10b64b P 38d6b9d0fc1649c0bc7c28d0f84bad9e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 04:46:59.936592 15324 tablet_bootstrap.cc:492] T efd1c3fda547461197fe9c044d10b64b P 38d6b9d0fc1649c0bc7c28d0f84bad9e: Bootstrap complete.
I20260925 04:46:59.938169 15324 ts_tablet_manager.cc:1397] T efd1c3fda547461197fe9c044d10b64b P 38d6b9d0fc1649c0bc7c28d0f84bad9e: Time spent bootstrapping tablet: real 0.032s user 0.022s sys 0.008s
I20260925 04:46:59.942866 15324 raft_consensus.cc:348] T efd1c3fda547461197fe9c044d10b64b P 38d6b9d0fc1649c0bc7c28d0f84bad9e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91628bf78d7a4e0b909f85c5b6af76a4" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43485 } } peers { permanent_uuid: "6f3e51d22a194e7db99512a63ba51e4f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 44125 } } peers { permanent_uuid: "38d6b9d0fc1649c0bc7c28d0f84bad9e" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41417 } }
I20260925 04:46:59.943635 15324 raft_consensus.cc:729] T efd1c3fda547461197fe9c044d10b64b P 38d6b9d0fc1649c0bc7c28d0f84bad9e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38d6b9d0fc1649c0bc7c28d0f84bad9e, State: Initialized, Role: FOLLOWER
I20260925 04:46:59.944355 15324 consensus_queue.cc:260] T efd1c3fda547461197fe9c044d10b64b P 38d6b9d0fc1649c0bc7c28d0f84bad9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91628bf78d7a4e0b909f85c5b6af76a4" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43485 } } peers { permanent_uuid: "6f3e51d22a194e7db99512a63ba51e4f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 44125 } } peers { permanent_uuid: "38d6b9d0fc1649c0bc7c28d0f84bad9e" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41417 } }
I20260925 04:46:59.945773 15324 ts_tablet_manager.cc:1428] T efd1c3fda547461197fe9c044d10b64b P 38d6b9d0fc1649c0bc7c28d0f84bad9e: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260925 04:46:59.978955 15434 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:46:59.921530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39037 (local address 127.0.98.65:57007)
0925 04:46:59.922147 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0925 04:46:59.922212 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0925 04:46:59.923598 (+ 1386us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:46:59.978003 (+ 54405us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39037: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":166,"threads_started":1}
W20260925 04:46:59.979568 15432 heartbeater.cc:644] Failed to heartbeat to 127.0.98.126:39037 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.126:39037: Client connection negotiation failed: client connection to 127.0.98.126:39037: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 04:46:59.980346 15434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:59.980085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39037 (local address 127.0.98.65:50717)
0925 04:46:59.980214 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 04:46:59.980283 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260925 04:46:59.981071 15434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:46:59.980878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39037 (local address 127.0.98.65:42707)
0925 04:46:59.980977 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0925 04:46:59.981020 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260925 04:46:59.981284 15432 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 04:47:00.056375 15436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:47:00.056699 15436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:47:00.056780 15436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:47:00.061080 15436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:47:00.061168 15436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:47:00.061268 15436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260925 04:47:00.067201 15436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:35919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:44125
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=33731
--tserver_master_addrs=127.0.98.126:39037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:47:00.068325 15436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:47:00.069362 15436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:47:00.079066 15443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:47:00.079226 15436 server_base.cc:1034] running on GCE node
W20260925 04:47:00.079043 15445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:47:00.079041 15442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:47:00.079888 15436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:47:00.080528 15436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:47:00.081724 15436 hybrid_clock.cc:648] HybridClock initialized: now 1790311620081690 us; error 54 us; skew 500 ppm
Sep 25 04:47:00 dist-test-slave-kghz krb5kdc[14122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311620, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:47:00.084633 15436 init.cc:377] Logged in from keytab as kudu/127.0.98.66@KRBTEST.COM (short username kudu)
I20260925 04:47:00.085513 15436 webserver.cc:460] Webserver started at http://127.0.98.66:33731/ using document root <none> and password file <none>
I20260925 04:47:00.086078 15436 fs_manager.cc:359] Metadata directory not provided
I20260925 04:47:00.086170 15436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:47:00.090058 15436 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260925 04:47:00.092353 15452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:47:00.093638 15436 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 04:47:00.093778 15436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-1/wal
uuid: "6f3e51d22a194e7db99512a63ba51e4f"
format_stamp: "Formatted at 2026-09-25 04:46:57 on dist-test-slave-kghz"
server_key: "e91bff69d701fad33b0c9d31d1e65f4d"
server_key_iv: "b250dfc98c43168455b1f65fa6b51e7b"
server_key_version: "encryptionkey@0"
I20260925 04:47:00.094403 15436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:47:00.104609 15436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:47:00.107939 15436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:47:00.108150 15436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:47:00.108762 15436 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:47:00.109584 15436 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:47:00.109660 15436 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:47:00.109733 15436 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:47:00.109781 15436 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:47:00.119481 15436 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:44125
I20260925 04:47:00.119505 15565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:44125 every 8 connection(s)
I20260925 04:47:00.120546 15436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
I20260925 04:47:00.121411 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 15436
I20260925 04:47:00.121789 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:43485
--local_ip_for_outbound_sockets=127.0.98.67
--tserver_master_addrs=127.0.98.126:39037
--webserver_port=39533
--webserver_interface=127.0.98.67
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:35919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 04:47:00.124858 15569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:00.122951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39037 (local address 127.0.98.66:33621)
0925 04:47:00.123452 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0925 04:47:00.123603 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":145,"threads_started":1}
W20260925 04:47:00.125519 15566 heartbeater.cc:644] Failed to heartbeat to 127.0.98.126:39037 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.126:39037: Client connection negotiation failed: client connection to 127.0.98.126:39037: connect: Connection refused (error 111)
I20260925 04:47:00.126315 15569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:00.126032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39037 (local address 127.0.98.66:42335)
0925 04:47:00.126155 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 04:47:00.126205 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260925 04:47:00.126991 15569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:00.126790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39037 (local address 127.0.98.66:49031)
0925 04:47:00.126898 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0925 04:47:00.126941 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 04:47:00.127175 15566 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 04:47:00.233194 15568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:47:00.233498 15568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:47:00.233574 15568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:47:00.239006 15568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:47:00.239094 15568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:47:00.239192 15568 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260925 04:47:00.244546 15568 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:35919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:43485
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=39533
--tserver_master_addrs=127.0.98.126:39037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:47:00.245709 15568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:47:00.246845 15568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 04:47:00.256533 15568 server_base.cc:1034] running on GCE node
W20260925 04:47:00.256459 15577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:47:00.256479 15575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:47:00.256474 15574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:47:00.257155 15568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:47:00.257745 15568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:47:00.258900 15568 hybrid_clock.cc:648] HybridClock initialized: now 1790311620258862 us; error 52 us; skew 500 ppm
Sep 25 04:47:00 dist-test-slave-kghz krb5kdc[14122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311620, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:47:00.261638 15568 init.cc:377] Logged in from keytab as kudu/127.0.98.67@KRBTEST.COM (short username kudu)
I20260925 04:47:00.262621 15568 webserver.cc:460] Webserver started at http://127.0.98.67:39533/ using document root <none> and password file <none>
I20260925 04:47:00.263180 15568 fs_manager.cc:359] Metadata directory not provided
I20260925 04:47:00.263293 15568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:47:00.267381 15568 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 04:47:00.269433 15584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:47:00.270917 15568 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 04:47:00.271015 15568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-2/wal
uuid: "91628bf78d7a4e0b909f85c5b6af76a4"
format_stamp: "Formatted at 2026-09-25 04:46:58 on dist-test-slave-kghz"
server_key: "91f46b945b0026e354cb60518ec43780"
server_key_iv: "e35f2d4b8d4c010ba7b881e9440673b5"
server_key_version: "encryptionkey@0"
I20260925 04:47:00.271581 15568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:47:00.287847 15568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:47:00.291137 15568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:47:00.291345 15568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:47:00.291962 15568 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:47:00.292774 15568 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:47:00.292820 15568 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:47:00.292886 15568 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:47:00.292918 15568 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:47:00.301920 15568 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:43485
I20260925 04:47:00.301949 15697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:43485 every 8 connection(s)
I20260925 04:47:00.302989 15568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
I20260925 04:47:00.306830 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:00.305070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39037 (local address 127.0.98.67:59537)
0925 04:47:00.305538 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0925 04:47:00.305651 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":116,"threads_started":1}
W20260925 04:47:00.307483 15698 heartbeater.cc:644] Failed to heartbeat to 127.0.98.126:39037 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.126:39037: Client connection negotiation failed: client connection to 127.0.98.126:39037: connect: Connection refused (error 111)
I20260925 04:47:00.308125 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:00.307912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39037 (local address 127.0.98.67:38955)
0925 04:47:00.308016 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0925 04:47:00.308069 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 04:47:00.308719 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 15568
I20260925 04:47:00.308750 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:00.308514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39037 (local address 127.0.98.67:54425)
0925 04:47:00.308636 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 04:47:00.308688 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260925 04:47:00.308939 15698 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 04:47:00.982300 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:00.981830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39037 (local address 127.0.98.65:46805)
0925 04:47:00.982075 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0925 04:47:00.982131 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":73,"threads_started":1}
I20260925 04:47:01.107038 15703 raft_consensus.cc:482] T efd1c3fda547461197fe9c044d10b64b P 38d6b9d0fc1649c0bc7c28d0f84bad9e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:47:01.107263 15703 raft_consensus.cc:504] T efd1c3fda547461197fe9c044d10b64b P 38d6b9d0fc1649c0bc7c28d0f84bad9e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91628bf78d7a4e0b909f85c5b6af76a4" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43485 } } peers { permanent_uuid: "6f3e51d22a194e7db99512a63ba51e4f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 44125 } } peers { permanent_uuid: "38d6b9d0fc1649c0bc7c28d0f84bad9e" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41417 } }
I20260925 04:47:01.108438 15703 leader_election.cc:288] T efd1c3fda547461197fe9c044d10b64b P 38d6b9d0fc1649c0bc7c28d0f84bad9e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 91628bf78d7a4e0b909f85c5b6af76a4 (127.0.98.67:43485), 6f3e51d22a194e7db99512a63ba51e4f (127.0.98.66:44125)
Sep 25 04:47:01 dist-test-slave-kghz krb5kdc[14122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311619, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for kudu/127.0.98.66@KRBTEST.COM
Sep 25 04:47:01 dist-test-slave-kghz krb5kdc[14122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311619, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for kudu/127.0.98.67@KRBTEST.COM
I20260925 04:47:01.122408 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:01.108658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:43485 (local address 127.0.98.65:33747)
0925 04:47:01.108774 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 04:47:01.108790 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:47:01.108894 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:47:01.110802 (+ 1908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:47:01.110807 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:47:01.111355 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:47:01.111822 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:47:01.111837 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:47:01.113085 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:47:01.113088 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:47:01.113936 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:47:01.113944 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:47:01.114700 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:47:01.114704 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:47:01.114931 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:47:01.115597 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:47:01.115623 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:47:01.117696 (+ 2073us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:47:01.120229 (+ 2533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:47:01.120232 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:47:01.120234 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:47:01.120457 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:47:01.120861 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:47:01.120864 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:47:01.120866 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:47:01.120954 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:47:01.121666 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:47:01.121669 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:47:01.121749 (+ 80us) client_negotiation.cc:769] Sending connection context
0925 04:47:01.121889 (+ 140us) client_negotiation.cc:240] Negotiation successful
0925 04:47:01.122215 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":86,"spinlock_wait_cycles":5248}
I20260925 04:47:01.122694 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:01.108816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:34009 (local address 127.0.98.66:44125)
0925 04:47:01.109166 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0925 04:47:01.109179 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 04:47:01.109216 (+ 37us) server_negotiation.cc:407] Connection header received
0925 04:47:01.109938 (+ 722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:47:01.109944 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:47:01.110390 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:47:01.110706 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:47:01.111991 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:47:01.112953 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:47:01.114068 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:47:01.114587 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:47:01.114748 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:47:01.117568 (+ 2820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:47:01.117574 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:47:01.117586 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:47:01.117617 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:47:01.119588 (+ 1971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:47:01.120091 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:47:01.120097 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:47:01.120112 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:47:01.120261 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:47:01.120690 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:47:01.120694 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:47:01.120695 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:47:01.121247 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:47:01.121361 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 04:47:01.122174 (+ 813us) server_negotiation.cc:299] Negotiation successful
0925 04:47:01.122474 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":93,"threads_started":1}
I20260925 04:47:01.123260 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:01.108664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:44125 (local address 127.0.98.65:34009)
0925 04:47:01.109106 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0925 04:47:01.109122 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:47:01.109223 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:47:01.110801 (+ 1578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:47:01.110807 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:47:01.111355 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:47:01.111822 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:47:01.111837 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:47:01.113080 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:47:01.113085 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:47:01.113947 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:47:01.113956 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:47:01.114703 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:47:01.114707 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:47:01.114937 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:47:01.115645 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:47:01.115657 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:47:01.117314 (+ 1657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:47:01.119696 (+ 2382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:47:01.119703 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:47:01.119722 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:47:01.119974 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:47:01.120384 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:47:01.120388 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:47:01.120391 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:47:01.120585 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:47:01.121427 (+ 842us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:47:01.121433 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:47:01.121702 (+ 269us) client_negotiation.cc:769] Sending connection context
0925 04:47:01.122010 (+ 308us) client_negotiation.cc:240] Negotiation successful
0925 04:47:01.123080 (+ 1070us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"mutex_wait_us":1148,"spinlock_wait_cycles":30336,"thread_start_us":69,"threads_started":1}
I20260925 04:47:01.123922 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:01.108813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:33747 (local address 127.0.98.67:43485)
0925 04:47:01.109310 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0925 04:47:01.109332 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 04:47:01.109362 (+ 30us) server_negotiation.cc:407] Connection header received
0925 04:47:01.109950 (+ 588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:47:01.109955 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:47:01.110393 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:47:01.110724 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:47:01.111991 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:47:01.112934 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:47:01.114055 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:47:01.114579 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:47:01.114748 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:47:01.117918 (+ 3170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:47:01.117927 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:47:01.117942 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:47:01.117977 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:47:01.120116 (+ 2139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:47:01.120575 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:47:01.120581 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:47:01.120586 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:47:01.120751 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:47:01.121052 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:47:01.121055 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:47:01.121057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:47:01.121546 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:47:01.122363 (+ 817us) server_negotiation.cc:1035] Waiting for connection context
0925 04:47:01.123442 (+ 1079us) server_negotiation.cc:299] Negotiation successful
0925 04:47:01.123743 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":138,"threads_started":1}
W20260925 04:47:01.125661 15320 leader_election.cc:341] T efd1c3fda547461197fe9c044d10b64b P 38d6b9d0fc1649c0bc7c28d0f84bad9e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6f3e51d22a194e7db99512a63ba51e4f (127.0.98.66:44125): Not found: Tablet not found: efd1c3fda547461197fe9c044d10b64b
W20260925 04:47:01.125911 15318 leader_election.cc:341] T efd1c3fda547461197fe9c044d10b64b P 38d6b9d0fc1649c0bc7c28d0f84bad9e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 91628bf78d7a4e0b909f85c5b6af76a4 (127.0.98.67:43485): Not found: Tablet not found: efd1c3fda547461197fe9c044d10b64b
I20260925 04:47:01.125984 15318 leader_election.cc:302] T efd1c3fda547461197fe9c044d10b64b P 38d6b9d0fc1649c0bc7c28d0f84bad9e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38d6b9d0fc1649c0bc7c28d0f84bad9e; no voters: 6f3e51d22a194e7db99512a63ba51e4f, 91628bf78d7a4e0b909f85c5b6af76a4
I20260925 04:47:01.126236 15703 raft_consensus.cc:2726] T efd1c3fda547461197fe9c044d10b64b P 38d6b9d0fc1649c0bc7c28d0f84bad9e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 04:47:01.128048 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:01.127682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39037 (local address 127.0.98.66:52519)
0925 04:47:01.127910 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0925 04:47:01.127961 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":65,"threads_started":1}
I20260925 04:47:01.309862 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:01.309464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39037 (local address 127.0.98.67:49921)
0925 04:47:01.309714 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0925 04:47:01.309766 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":71,"threads_started":1}
I20260925 04:47:01.983590 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:01.983148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39037 (local address 127.0.98.65:58299)
0925 04:47:01.983423 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0925 04:47:01.983491 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":80,"threads_started":1}
I20260925 04:47:02.129212 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:02.128793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39037 (local address 127.0.98.66:50769)
0925 04:47:02.129044 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0925 04:47:02.129118 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":75,"threads_started":1}
I20260925 04:47:02.311208 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:02.310775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39037 (local address 127.0.98.67:57121)
0925 04:47:02.311032 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0925 04:47:02.311094 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":75,"threads_started":1}
I20260925 04:47:02.984830 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:02.984385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39037 (local address 127.0.98.65:46197)
0925 04:47:02.984681 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0925 04:47:02.984733 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":74,"threads_started":1}
I20260925 04:47:03.115182 15713 raft_consensus.cc:482] T efd1c3fda547461197fe9c044d10b64b P 38d6b9d0fc1649c0bc7c28d0f84bad9e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:47:03.115420 15713 raft_consensus.cc:504] T efd1c3fda547461197fe9c044d10b64b P 38d6b9d0fc1649c0bc7c28d0f84bad9e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91628bf78d7a4e0b909f85c5b6af76a4" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43485 } } peers { permanent_uuid: "6f3e51d22a194e7db99512a63ba51e4f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 44125 } } peers { permanent_uuid: "38d6b9d0fc1649c0bc7c28d0f84bad9e" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41417 } }
I20260925 04:47:03.115949 15713 leader_election.cc:288] T efd1c3fda547461197fe9c044d10b64b P 38d6b9d0fc1649c0bc7c28d0f84bad9e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 91628bf78d7a4e0b909f85c5b6af76a4 (127.0.98.67:43485), 6f3e51d22a194e7db99512a63ba51e4f (127.0.98.66:44125)
W20260925 04:47:03.116707 15320 leader_election.cc:341] T efd1c3fda547461197fe9c044d10b64b P 38d6b9d0fc1649c0bc7c28d0f84bad9e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6f3e51d22a194e7db99512a63ba51e4f (127.0.98.66:44125): Not found: Tablet not found: efd1c3fda547461197fe9c044d10b64b
W20260925 04:47:03.116796 15318 leader_election.cc:341] T efd1c3fda547461197fe9c044d10b64b P 38d6b9d0fc1649c0bc7c28d0f84bad9e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 91628bf78d7a4e0b909f85c5b6af76a4 (127.0.98.67:43485): Not found: Tablet not found: efd1c3fda547461197fe9c044d10b64b
I20260925 04:47:03.116902 15318 leader_election.cc:302] T efd1c3fda547461197fe9c044d10b64b P 38d6b9d0fc1649c0bc7c28d0f84bad9e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38d6b9d0fc1649c0bc7c28d0f84bad9e; no voters: 6f3e51d22a194e7db99512a63ba51e4f, 91628bf78d7a4e0b909f85c5b6af76a4
I20260925 04:47:03.117059 15713 raft_consensus.cc:2726] T efd1c3fda547461197fe9c044d10b64b P 38d6b9d0fc1649c0bc7c28d0f84bad9e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 04:47:03.130504 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:03.130002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39037 (local address 127.0.98.66:42341)
0925 04:47:03.130336 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0925 04:47:03.130396 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":88,"threads_started":1}
I20260925 04:47:03.312391 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:03.311979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39037 (local address 127.0.98.67:37867)
0925 04:47:03.312243 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0925 04:47:03.312298 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":90,"threads_started":1}
I20260925 04:47:03.986059 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:03.985628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39037 (local address 127.0.98.65:55503)
0925 04:47:03.985888 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0925 04:47:03.985962 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":79,"threads_started":1}
I20260925 04:47:04.131783 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:04.131336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39037 (local address 127.0.98.66:59253)
0925 04:47:04.131622 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0925 04:47:04.131685 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":76,"threads_started":1}
I20260925 04:47:04.313603 15718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:04.313157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39037 (local address 127.0.98.67:44253)
0925 04:47:04.313436 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0925 04:47:04.313495 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":87,"threads_started":1}
I20260925 04:47:04.987612 15719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:04.987052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39037 (local address 127.0.98.65:58117)
0925 04:47:04.987416 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0925 04:47:04.987480 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":78,"threads_started":1}
I20260925 04:47:05.133052 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:05.132626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39037 (local address 127.0.98.66:39593)
0925 04:47:05.132899 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0925 04:47:05.132956 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":91,"threads_started":1}
I20260925 04:47:05.314960 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:05.314472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39037 (local address 127.0.98.67:33597)
0925 04:47:05.314801 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0925 04:47:05.314862 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":108,"threads_started":1}
I20260925 04:47:05.988971 15722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:05.988507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39037 (local address 127.0.98.65:34979)
0925 04:47:05.988807 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0925 04:47:05.988871 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":78,"threads_started":1}
I20260925 04:47:06.134305 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:06.133816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39037 (local address 127.0.98.66:45441)
0925 04:47:06.134085 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0925 04:47:06.134149 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":71,"threads_started":1}
I20260925 04:47:06.160460 15724 raft_consensus.cc:482] T efd1c3fda547461197fe9c044d10b64b P 38d6b9d0fc1649c0bc7c28d0f84bad9e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:47:06.160593 15724 raft_consensus.cc:504] T efd1c3fda547461197fe9c044d10b64b P 38d6b9d0fc1649c0bc7c28d0f84bad9e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91628bf78d7a4e0b909f85c5b6af76a4" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43485 } } peers { permanent_uuid: "6f3e51d22a194e7db99512a63ba51e4f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 44125 } } peers { permanent_uuid: "38d6b9d0fc1649c0bc7c28d0f84bad9e" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41417 } }
I20260925 04:47:06.161029 15724 leader_election.cc:288] T efd1c3fda547461197fe9c044d10b64b P 38d6b9d0fc1649c0bc7c28d0f84bad9e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 91628bf78d7a4e0b909f85c5b6af76a4 (127.0.98.67:43485), 6f3e51d22a194e7db99512a63ba51e4f (127.0.98.66:44125)
W20260925 04:47:06.161818 15318 leader_election.cc:341] T efd1c3fda547461197fe9c044d10b64b P 38d6b9d0fc1649c0bc7c28d0f84bad9e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 91628bf78d7a4e0b909f85c5b6af76a4 (127.0.98.67:43485): Not found: Tablet not found: efd1c3fda547461197fe9c044d10b64b
W20260925 04:47:06.161928 15320 leader_election.cc:341] T efd1c3fda547461197fe9c044d10b64b P 38d6b9d0fc1649c0bc7c28d0f84bad9e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6f3e51d22a194e7db99512a63ba51e4f (127.0.98.66:44125): Not found: Tablet not found: efd1c3fda547461197fe9c044d10b64b
I20260925 04:47:06.162119 15318 leader_election.cc:302] T efd1c3fda547461197fe9c044d10b64b P 38d6b9d0fc1649c0bc7c28d0f84bad9e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38d6b9d0fc1649c0bc7c28d0f84bad9e; no voters: 6f3e51d22a194e7db99512a63ba51e4f, 91628bf78d7a4e0b909f85c5b6af76a4
I20260925 04:47:06.162297 15724 raft_consensus.cc:2726] T efd1c3fda547461197fe9c044d10b64b P 38d6b9d0fc1649c0bc7c28d0f84bad9e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 04:47:06.316176 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:06.315731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39037 (local address 127.0.98.67:46501)
0925 04:47:06.316017 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0925 04:47:06.316077 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":93,"threads_started":1}
I20260925 04:47:06.990489 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:06.989935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39037 (local address 127.0.98.65:38581)
0925 04:47:06.990220 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0925 04:47:06.990369 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":81,"threads_started":1}
I20260925 04:47:07.135711 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:07.135208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39037 (local address 127.0.98.66:36029)
0925 04:47:07.135547 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0925 04:47:07.135607 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":101,"threads_started":1}
I20260925 04:47:07.317413 15728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:07.316941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39037 (local address 127.0.98.67:58999)
0925 04:47:07.317261 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0925 04:47:07.317318 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":85,"threads_started":1}
I20260925 04:47:07.991806 15729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:07.991372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39037 (local address 127.0.98.65:56535)
0925 04:47:07.991648 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0925 04:47:07.991708 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":75,"threads_started":1}
I20260925 04:47:08.136988 15730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:08.136527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39037 (local address 127.0.98.66:43725)
0925 04:47:08.136820 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0925 04:47:08.136887 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":91,"threads_started":1}
I20260925 04:47:08.318696 15731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:08.318220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39037 (local address 127.0.98.67:55891)
0925 04:47:08.318536 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0925 04:47:08.318593 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":95,"threads_started":1}
I20260925 04:47:08.888413 15732 raft_consensus.cc:482] T efd1c3fda547461197fe9c044d10b64b P 38d6b9d0fc1649c0bc7c28d0f84bad9e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:47:08.888540 15732 raft_consensus.cc:504] T efd1c3fda547461197fe9c044d10b64b P 38d6b9d0fc1649c0bc7c28d0f84bad9e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91628bf78d7a4e0b909f85c5b6af76a4" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43485 } } peers { permanent_uuid: "6f3e51d22a194e7db99512a63ba51e4f" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 44125 } } peers { permanent_uuid: "38d6b9d0fc1649c0bc7c28d0f84bad9e" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41417 } }
I20260925 04:47:08.888959 15732 leader_election.cc:288] T efd1c3fda547461197fe9c044d10b64b P 38d6b9d0fc1649c0bc7c28d0f84bad9e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 91628bf78d7a4e0b909f85c5b6af76a4 (127.0.98.67:43485), 6f3e51d22a194e7db99512a63ba51e4f (127.0.98.66:44125)
W20260925 04:47:08.889807 15320 leader_election.cc:341] T efd1c3fda547461197fe9c044d10b64b P 38d6b9d0fc1649c0bc7c28d0f84bad9e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6f3e51d22a194e7db99512a63ba51e4f (127.0.98.66:44125): Not found: Tablet not found: efd1c3fda547461197fe9c044d10b64b
W20260925 04:47:08.889918 15318 leader_election.cc:341] T efd1c3fda547461197fe9c044d10b64b P 38d6b9d0fc1649c0bc7c28d0f84bad9e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 91628bf78d7a4e0b909f85c5b6af76a4 (127.0.98.67:43485): Not found: Tablet not found: efd1c3fda547461197fe9c044d10b64b
I20260925 04:47:08.890028 15318 leader_election.cc:302] T efd1c3fda547461197fe9c044d10b64b P 38d6b9d0fc1649c0bc7c28d0f84bad9e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38d6b9d0fc1649c0bc7c28d0f84bad9e; no voters: 6f3e51d22a194e7db99512a63ba51e4f, 91628bf78d7a4e0b909f85c5b6af76a4
I20260925 04:47:08.890199 15732 raft_consensus.cc:2726] T efd1c3fda547461197fe9c044d10b64b P 38d6b9d0fc1649c0bc7c28d0f84bad9e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 04:47:08.993229 15733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:08.992753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39037 (local address 127.0.98.65:49209)
0925 04:47:08.993038 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0925 04:47:08.993120 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":80,"threads_started":1}
I20260925 04:47:09.138300 15734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:09.137787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39037 (local address 127.0.98.66:37363)
0925 04:47:09.138073 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0925 04:47:09.138147 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":98,"threads_started":1}
I20260925 04:47:09.319913 15735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:09.319443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39037 (local address 127.0.98.67:59873)
0925 04:47:09.319732 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0925 04:47:09.319810 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":77,"threads_started":1}
I20260925 04:47:09.994675 15736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:09.994121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39037 (local address 127.0.98.65:56537)
0925 04:47:09.994509 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0925 04:47:09.994576 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":76,"threads_started":1}
I20260925 04:47:10.139686 15737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:10.139200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39037 (local address 127.0.98.66:42493)
0925 04:47:10.139517 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0925 04:47:10.139584 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":100,"threads_started":1}
I20260925 04:47:10.321199 15738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:10.320779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39037 (local address 127.0.98.67:55699)
0925 04:47:10.321043 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0925 04:47:10.321102 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":104,"threads_started":1}
I20260925 04:47:10.996070 15739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:10.995536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39037 (local address 127.0.98.65:33387)
0925 04:47:10.995836 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0925 04:47:10.995898 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":105,"threads_started":1}
I20260925 04:47:11.140969 15740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:11.140527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39037 (local address 127.0.98.66:36545)
0925 04:47:11.140801 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0925 04:47:11.140867 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":78,"threads_started":1}
I20260925 04:47:11.322508 15741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:11.322075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39037 (local address 127.0.98.67:45813)
0925 04:47:11.322346 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0925 04:47:11.322408 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":77,"threads_started":1}
I20260925 04:47:11.997355 15742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:11.996911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39037 (local address 127.0.98.65:42015)
0925 04:47:11.997202 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0925 04:47:11.997259 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":97,"threads_started":1}
I20260925 04:47:12.142474 15743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:12.141960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39037 (local address 127.0.98.66:60433)
0925 04:47:12.142302 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0925 04:47:12.142369 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":94,"threads_started":1}
I20260925 04:47:12.323967 15744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:12.323511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39037 (local address 127.0.98.67:57649)
0925 04:47:12.323801 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0925 04:47:12.323866 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":96,"threads_started":1}
I20260925 04:47:12.998952 15745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:12.998486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39037 (local address 127.0.98.65:48513)
0925 04:47:12.998779 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0925 04:47:12.998849 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":94,"threads_started":1}
I20260925 04:47:13.143817 15746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:13.143333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39037 (local address 127.0.98.66:35041)
0925 04:47:13.143642 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0925 04:47:13.143708 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":77,"threads_started":1}
I20260925 04:47:13.325201 15747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:13.324768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39037 (local address 127.0.98.67:58463)
0925 04:47:13.325032 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0925 04:47:13.325092 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":91,"threads_started":1}
I20260925 04:47:14.000262 15748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:13.999833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39037 (local address 127.0.98.65:52053)
0925 04:47:14.000100 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0925 04:47:14.000163 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":103,"threads_started":1}
I20260925 04:47:14.145063 15749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:14.144613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39037 (local address 127.0.98.66:59747)
0925 04:47:14.144914 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0925 04:47:14.144966 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":120,"threads_started":1}
I20260925 04:47:14.326578 15750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:14.326052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39037 (local address 127.0.98.67:55471)
0925 04:47:14.326414 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0925 04:47:14.326479 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":74,"threads_started":1}
I20260925 04:47:15.001951 15751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:15.001310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39037 (local address 127.0.98.65:48057)
0925 04:47:15.001710 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0925 04:47:15.001797 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":94,"threads_started":1}
I20260925 04:47:15.146407 15752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:15.145939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39037 (local address 127.0.98.66:40999)
0925 04:47:15.146180 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0925 04:47:15.146293 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":75,"threads_started":1}
I20260925 04:47:15.319950 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 15268
I20260925 04:47:15.325865 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 15436
I20260925 04:47:15.328152 15753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:15.327622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39037 (local address 127.0.98.67:34287)
0925 04:47:15.327923 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0925 04:47:15.327998 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":65,"threads_started":1}
I20260925 04:47:15.331542 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 15568
2026-09-25T04:47:15Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19404 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:47:15 dist-test-slave-kghz krb5kdc[15756](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:47:15 dist-test-slave-kghz krb5kdc[15756](info): set up 2 sockets
Sep 25 04:47:15 dist-test-slave-kghz krb5kdc[15756](info): commencing operation
krb5kdc: starting...
W20260925 04:47:16.386623 393 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/krb5kdc/test-user.keytab.
Sep 25 04:47:16 dist-test-slave-kghz krb5kdc[15756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:47:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:47:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260925 04:47:16.517230 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:33181
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:39863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:33181
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:47:16.629848 15769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:47:16.630230 15769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:47:16.630338 15769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:47:16.634497 15769 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 04:47:16.634599 15769 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 04:47:16.634634 15769 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 04:47:16.634666 15769 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:47:16.634711 15769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:47:16.634757 15769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:47:16.634814 15769 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:47:16.634855 15769 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:47:16.640415 15769 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:39863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:33181
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:33181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15769
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:47:16.641615 15769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:47:16.642724 15769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:47:16.651475 15777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:47:16.651468 15775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:47:16.651463 15774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:47:16.651721 15769 server_base.cc:1034] running on GCE node
I20260925 04:47:16.652382 15769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:47:16.653587 15769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:47:16.654819 15769 hybrid_clock.cc:648] HybridClock initialized: now 1790311636654798 us; error 36 us; skew 500 ppm
Sep 25 04:47:16 dist-test-slave-kghz krb5kdc[15756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311636, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:47:16.657708 15769 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260925 04:47:16.658700 15769 webserver.cc:460] Webserver started at http://127.0.98.126:38533/ using document root <none> and password file <none>
I20260925 04:47:16.659329 15769 fs_manager.cc:359] Metadata directory not provided
I20260925 04:47:16.659394 15769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:47:16.659574 15769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:47:16.661654 15769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/master-0/data/instance:
uuid: "a175a4ff2ea44c5ca7a8f69fec22566a"
format_stamp: "Formatted at 2026-09-25 04:47:16 on dist-test-slave-kghz"
server_key: "184464def56b690e2a70b606620f3760"
server_key_iv: "b9c5f2a8beaab36785dae5e8815eeff3"
server_key_version: "encryptionkey@0"
I20260925 04:47:16.662212 15769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/master-0/wal/instance:
uuid: "a175a4ff2ea44c5ca7a8f69fec22566a"
format_stamp: "Formatted at 2026-09-25 04:47:16 on dist-test-slave-kghz"
server_key: "184464def56b690e2a70b606620f3760"
server_key_iv: "b9c5f2a8beaab36785dae5e8815eeff3"
server_key_version: "encryptionkey@0"
I20260925 04:47:16.666231 15769 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 04:47:16.669018 15784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:47:16.670404 15769 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260925 04:47:16.670547 15769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/master-0/wal
uuid: "a175a4ff2ea44c5ca7a8f69fec22566a"
format_stamp: "Formatted at 2026-09-25 04:47:16 on dist-test-slave-kghz"
server_key: "184464def56b690e2a70b606620f3760"
server_key_iv: "b9c5f2a8beaab36785dae5e8815eeff3"
server_key_version: "encryptionkey@0"
I20260925 04:47:16.670666 15769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:47:16.689355 15769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:47:16.692622 15769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:47:16.692840 15769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:47:16.700176 15769 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:33181
I20260925 04:47:16.700489 15836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:33181 every 8 connection(s)
I20260925 04:47:16.701231 15769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
I20260925 04:47:16.704061 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 15769
I20260925 04:47:16.704223 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/master-0/wal/instance
I20260925 04:47:16.704406 15837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:47:16.704577 393 external_mini_cluster.cc:1428] Setting key 326e4ef4df414324005a9c2c48251d4a
I20260925 04:47:16.712049 15837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a175a4ff2ea44c5ca7a8f69fec22566a: Bootstrap starting.
Sep 25 04:47:16 dist-test-slave-kghz krb5kdc[15756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:47:16.714804 15837 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a175a4ff2ea44c5ca7a8f69fec22566a: Neither blocks nor log segments found. Creating new log.
I20260925 04:47:16.715811 15837 log.cc:824] T 00000000000000000000000000000000 P a175a4ff2ea44c5ca7a8f69fec22566a: Log is configured to *not* fsync() on all Append() calls
I20260925 04:47:16.718343 15837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a175a4ff2ea44c5ca7a8f69fec22566a: No bootstrap required, opened a new log
I20260925 04:47:16.720453 15840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:16.705994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36836 (local address 127.0.98.126:33181)
0925 04:47:16.706486 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0925 04:47:16.706507 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 04:47:16.706590 (+ 83us) server_negotiation.cc:407] Connection header received
0925 04:47:16.707420 (+ 830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:47:16.707447 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:47:16.707900 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:47:16.708315 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:47:16.708973 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:47:16.710517 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:47:16.711394 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:47:16.712077 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:47:16.712252 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:47:16.715223 (+ 2971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:47:16.715229 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:47:16.715247 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:47:16.715282 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:47:16.717305 (+ 2023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:47:16.717778 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:47:16.717784 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:47:16.717790 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:47:16.717929 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:47:16.718200 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:47:16.718203 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:47:16.718205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:47:16.718765 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:47:16.718946 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0925 04:47:16.719196 (+ 250us) server_negotiation.cc:299] Negotiation successful
0925 04:47:16.719492 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":138,"threads_started":1}
I20260925 04:47:16.722855 15837 raft_consensus.cc:348] T 00000000000000000000000000000000 P a175a4ff2ea44c5ca7a8f69fec22566a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a175a4ff2ea44c5ca7a8f69fec22566a" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33181 } }
I20260925 04:47:16.723109 15837 raft_consensus.cc:374] T 00000000000000000000000000000000 P a175a4ff2ea44c5ca7a8f69fec22566a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:47:16.723155 15837 raft_consensus.cc:729] T 00000000000000000000000000000000 P a175a4ff2ea44c5ca7a8f69fec22566a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a175a4ff2ea44c5ca7a8f69fec22566a, State: Initialized, Role: FOLLOWER
I20260925 04:47:16.723568 15837 consensus_queue.cc:260] T 00000000000000000000000000000000 P a175a4ff2ea44c5ca7a8f69fec22566a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a175a4ff2ea44c5ca7a8f69fec22566a" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33181 } }
I20260925 04:47:16.723686 15837 raft_consensus.cc:388] T 00000000000000000000000000000000 P a175a4ff2ea44c5ca7a8f69fec22566a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:47:16.723726 15837 raft_consensus.cc:482] T 00000000000000000000000000000000 P a175a4ff2ea44c5ca7a8f69fec22566a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:47:16.723796 15837 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a175a4ff2ea44c5ca7a8f69fec22566a [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:47:16.724663 15837 raft_consensus.cc:504] T 00000000000000000000000000000000 P a175a4ff2ea44c5ca7a8f69fec22566a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a175a4ff2ea44c5ca7a8f69fec22566a" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33181 } }
I20260925 04:47:16.724951 15837 leader_election.cc:302] T 00000000000000000000000000000000 P a175a4ff2ea44c5ca7a8f69fec22566a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a175a4ff2ea44c5ca7a8f69fec22566a; no voters:
I20260925 04:47:16.725206 15837 leader_election.cc:288] T 00000000000000000000000000000000 P a175a4ff2ea44c5ca7a8f69fec22566a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:47:16.725353 15842 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a175a4ff2ea44c5ca7a8f69fec22566a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:47:16.725644 15842 raft_consensus.cc:686] T 00000000000000000000000000000000 P a175a4ff2ea44c5ca7a8f69fec22566a [term 1 LEADER]: Becoming Leader. State: Replica: a175a4ff2ea44c5ca7a8f69fec22566a, State: Running, Role: LEADER
I20260925 04:47:16.725944 15842 consensus_queue.cc:237] T 00000000000000000000000000000000 P a175a4ff2ea44c5ca7a8f69fec22566a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a175a4ff2ea44c5ca7a8f69fec22566a" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33181 } }
I20260925 04:47:16.726514 15837 sys_catalog.cc:564] T 00000000000000000000000000000000 P a175a4ff2ea44c5ca7a8f69fec22566a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:47:16.727767 15843 sys_catalog.cc:455] T 00000000000000000000000000000000 P a175a4ff2ea44c5ca7a8f69fec22566a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a175a4ff2ea44c5ca7a8f69fec22566a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a175a4ff2ea44c5ca7a8f69fec22566a" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33181 } } }
I20260925 04:47:16.727810 15844 sys_catalog.cc:455] T 00000000000000000000000000000000 P a175a4ff2ea44c5ca7a8f69fec22566a [sys.catalog]: SysCatalogTable state changed. Reason: New leader a175a4ff2ea44c5ca7a8f69fec22566a. Latest consensus state: current_term: 1 leader_uuid: "a175a4ff2ea44c5ca7a8f69fec22566a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a175a4ff2ea44c5ca7a8f69fec22566a" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33181 } } }
I20260925 04:47:16.727906 15844 sys_catalog.cc:458] T 00000000000000000000000000000000 P a175a4ff2ea44c5ca7a8f69fec22566a [sys.catalog]: This master's current role is: LEADER
I20260925 04:47:16.727906 15843 sys_catalog.cc:458] T 00000000000000000000000000000000 P a175a4ff2ea44c5ca7a8f69fec22566a [sys.catalog]: This master's current role is: LEADER
I20260925 04:47:16.728497 15852 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:47:16.731932 15852 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:47:16.737764 15852 catalog_manager.cc:1347] Generated new cluster ID: c272a46e210846a7a5d01e8e195412a3
I20260925 04:47:16.737845 15852 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:47:16.756510 15852 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:47:16.757931 15852 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:47:16.767774 15852 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a175a4ff2ea44c5ca7a8f69fec22566a: Generated new TSK 0
I20260925 04:47:16.768539 15852 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/krb5kdc/kudu_127.0.98.65.keytab.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/krb5kdc/kudu_127.0.98.65.keytab.
I20260925 04:47:16.806453 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:0
--local_ip_for_outbound_sockets=127.0.98.65
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:33181
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:39863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:47:16.927971 15863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:47:16.928242 15863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:47:16.928287 15863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:47:16.932376 15863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:47:16.932488 15863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:47:16.932607 15863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260925 04:47:16.937884 15863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:39863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:33181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15863
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:47:16.939003 15863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:47:16.940061 15863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 04:47:16.949900 15863 server_base.cc:1034] running on GCE node
W20260925 04:47:16.949836 15869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:47:16.949920 15868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:47:16.949836 15871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:47:16.950604 15863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:47:16.951261 15863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:47:16.952461 15863 hybrid_clock.cc:648] HybridClock initialized: now 1790311636952427 us; error 47 us; skew 500 ppm
Sep 25 04:47:16 dist-test-slave-kghz krb5kdc[15756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311636, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:47:16.955456 15863 init.cc:377] Logged in from keytab as kudu/127.0.98.65@KRBTEST.COM (short username kudu)
I20260925 04:47:16.956363 15863 webserver.cc:460] Webserver started at http://127.0.98.65:38693/ using document root <none> and password file <none>
I20260925 04:47:16.956945 15863 fs_manager.cc:359] Metadata directory not provided
I20260925 04:47:16.957053 15863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:47:16.957298 15863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:47:16.959467 15863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-0/data/instance:
uuid: "7a05a5d169f8498a823f5f8b7ac15f92"
format_stamp: "Formatted at 2026-09-25 04:47:16 on dist-test-slave-kghz"
server_key: "42d5f01b4c6f1a6ccbf7cd191e555c7d"
server_key_iv: "b2d33c06103ea5639fe80b0a86c5cfd6"
server_key_version: "encryptionkey@0"
I20260925 04:47:16.960095 15863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-0/wal/instance:
uuid: "7a05a5d169f8498a823f5f8b7ac15f92"
format_stamp: "Formatted at 2026-09-25 04:47:16 on dist-test-slave-kghz"
server_key: "42d5f01b4c6f1a6ccbf7cd191e555c7d"
server_key_iv: "b2d33c06103ea5639fe80b0a86c5cfd6"
server_key_version: "encryptionkey@0"
I20260925 04:47:16.963825 15863 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 04:47:16.966167 15878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:47:16.967473 15863 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 04:47:16.967628 15863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-0/wal
uuid: "7a05a5d169f8498a823f5f8b7ac15f92"
format_stamp: "Formatted at 2026-09-25 04:47:16 on dist-test-slave-kghz"
server_key: "42d5f01b4c6f1a6ccbf7cd191e555c7d"
server_key_iv: "b2d33c06103ea5639fe80b0a86c5cfd6"
server_key_version: "encryptionkey@0"
I20260925 04:47:16.967746 15863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:47:16.988984 15863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:47:16.992138 15863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:47:16.992350 15863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:47:16.992974 15863 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:47:16.993829 15863 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:47:16.993902 15863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:47:16.993978 15863 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:47:16.994021 15863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:47:17.002959 15863 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:41669
I20260925 04:47:17.003001 15991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:41669 every 8 connection(s)
I20260925 04:47:17.003887 15863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
Sep 25 04:47:17 dist-test-slave-kghz krb5kdc[15756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311636, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:47:17.013916 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 15863
I20260925 04:47:17.014074 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-0/wal/instance
I20260925 04:47:17.014448 393 external_mini_cluster.cc:1428] Setting key 68ffda3166453046e1dde733347f7657
I20260925 04:47:17.018236 15840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:17.005746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:53109 (local address 127.0.98.126:33181)
0925 04:47:17.005901 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 04:47:17.005906 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:47:17.007064 (+ 1158us) server_negotiation.cc:407] Connection header received
0925 04:47:17.007945 (+ 881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:47:17.007948 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:47:17.007993 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:47:17.008076 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:47:17.009292 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:47:17.009863 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:47:17.011051 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:47:17.011291 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:47:17.011378 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:47:17.014107 (+ 2729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:47:17.014128 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:47:17.014133 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:47:17.014168 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:47:17.015838 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:47:17.016422 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:47:17.016425 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:47:17.016426 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:47:17.016501 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:47:17.016899 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:47:17.016901 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:47:17.016902 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:47:17.017078 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:47:17.017152 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 04:47:17.017851 (+ 699us) server_negotiation.cc:299] Negotiation successful
0925 04:47:17.018000 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 04:47:17.018997 15994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:17.006166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:33181 (local address 127.0.98.65:53109)
0925 04:47:17.006915 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0925 04:47:17.006958 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 04:47:17.007766 (+ 808us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:47:17.008267 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:47:17.008274 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:47:17.008633 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:47:17.009148 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:47:17.009161 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:47:17.009982 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:47:17.009985 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:47:17.010939 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:47:17.010949 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:47:17.011492 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:47:17.011495 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:47:17.011645 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:47:17.012318 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:47:17.012334 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:47:17.013865 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:47:17.015947 (+ 2082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:47:17.015953 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:47:17.015969 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:47:17.016321 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:47:17.016591 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:47:17.016597 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:47:17.016602 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:47:17.016808 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:47:17.017177 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:47:17.017187 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:47:17.017544 (+ 357us) client_negotiation.cc:769] Sending connection context
0925 04:47:17.017771 (+ 227us) client_negotiation.cc:240] Negotiation successful
0925 04:47:17.018045 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":132,"threads_started":1}
I20260925 04:47:17.020239 15992 heartbeater.cc:347] Connected to a master server at 127.0.98.126:33181
I20260925 04:47:17.020509 15992 heartbeater.cc:464] Registering TS with master...
I20260925 04:47:17.021148 15992 heartbeater.cc:511] Master 127.0.98.126:33181 requested a full tablet report, sending...
I20260925 04:47:17.022815 15801 ts_manager.cc:194] Registered new tserver with Master: 7a05a5d169f8498a823f5f8b7ac15f92 (127.0.98.65:41669)
I20260925 04:47:17.023953 15801 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.65@KRBTEST.COM'} at 127.0.98.65:53109
WARNING: no policy specified for kudu/127.0.98.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/krb5kdc/kudu_127.0.98.66.keytab.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/krb5kdc/kudu_127.0.98.66.keytab.
I20260925 04:47:17.041555 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:0
--local_ip_for_outbound_sockets=127.0.98.66
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:33181
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:39863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:47:17.152508 15997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:47:17.152792 15997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:47:17.152871 15997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:47:17.157116 15997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:47:17.157196 15997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:47:17.157295 15997 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260925 04:47:17.163141 15997 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:39863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:33181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15997
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:47:17.164275 15997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:47:17.165336 15997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:47:17.175390 16005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:47:17.175414 16003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:47:17.175401 16002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:47:17.175632 15997 server_base.cc:1034] running on GCE node
I20260925 04:47:17.176141 15997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:47:17.177006 15997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:47:17.178207 15997 hybrid_clock.cc:648] HybridClock initialized: now 1790311637178114 us; error 127 us; skew 500 ppm
Sep 25 04:47:17 dist-test-slave-kghz krb5kdc[15756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311637, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:47:17.181264 15997 init.cc:377] Logged in from keytab as kudu/127.0.98.66@KRBTEST.COM (short username kudu)
I20260925 04:47:17.182240 15997 webserver.cc:460] Webserver started at http://127.0.98.66:33783/ using document root <none> and password file <none>
I20260925 04:47:17.182911 15997 fs_manager.cc:359] Metadata directory not provided
I20260925 04:47:17.183008 15997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:47:17.183259 15997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:47:17.185389 15997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-1/data/instance:
uuid: "c74e7ff9fd45472cbcc9cb1f76994e3e"
format_stamp: "Formatted at 2026-09-25 04:47:17 on dist-test-slave-kghz"
server_key: "f957305e70af52a37fc92da714a64437"
server_key_iv: "9502451580e9c1b30f9ee1a05a402086"
server_key_version: "encryptionkey@0"
I20260925 04:47:17.185990 15997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-1/wal/instance:
uuid: "c74e7ff9fd45472cbcc9cb1f76994e3e"
format_stamp: "Formatted at 2026-09-25 04:47:17 on dist-test-slave-kghz"
server_key: "f957305e70af52a37fc92da714a64437"
server_key_iv: "9502451580e9c1b30f9ee1a05a402086"
server_key_version: "encryptionkey@0"
I20260925 04:47:17.189883 15997 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 04:47:17.192461 16012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:47:17.193693 15997 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 04:47:17.193841 15997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-1/wal
uuid: "c74e7ff9fd45472cbcc9cb1f76994e3e"
format_stamp: "Formatted at 2026-09-25 04:47:17 on dist-test-slave-kghz"
server_key: "f957305e70af52a37fc92da714a64437"
server_key_iv: "9502451580e9c1b30f9ee1a05a402086"
server_key_version: "encryptionkey@0"
I20260925 04:47:17.193972 15997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:47:17.211705 15997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:47:17.215170 15997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:47:17.215399 15997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:47:17.216033 15997 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:47:17.216909 15997 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:47:17.216987 15997 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:47:17.217061 15997 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:47:17.217105 15997 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:47:17.226588 15997 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:43903
I20260925 04:47:17.226651 16125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:43903 every 8 connection(s)
I20260925 04:47:17.227641 15997 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
I20260925 04:47:17.228032 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 15997
I20260925 04:47:17.228168 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-1/wal/instance
I20260925 04:47:17.228509 393 external_mini_cluster.cc:1428] Setting key d37d1a745a85788955e3078d3e8c6e1d
WARNING: no policy specified for kudu/127.0.98.67@KRBTEST.COM; defaulting to no policy
Sep 25 04:47:17 dist-test-slave-kghz krb5kdc[15756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311637, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.67@KRBTEST.COM" created.
I20260925 04:47:17.243058 15840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:17.229613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:49493 (local address 127.0.98.126:33181)
0925 04:47:17.229750 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 04:47:17.229754 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:47:17.231076 (+ 1322us) server_negotiation.cc:407] Connection header received
0925 04:47:17.232424 (+ 1348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:47:17.232429 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:47:17.232495 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:47:17.232614 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:47:17.233821 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:47:17.234818 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:47:17.235979 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:47:17.236366 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:47:17.236492 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:47:17.239145 (+ 2653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:47:17.239148 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:47:17.239150 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:47:17.239174 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:47:17.240711 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:47:17.241329 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:47:17.241335 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:47:17.241340 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:47:17.241434 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:47:17.241864 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:47:17.241866 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:47:17.241868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:47:17.242047 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:47:17.242126 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 04:47:17.242823 (+ 697us) server_negotiation.cc:299] Negotiation successful
0925 04:47:17.242930 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 04:47:17.244364 16128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:17.230050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:33181 (local address 127.0.98.66:49493)
0925 04:47:17.230882 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0925 04:47:17.230947 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0925 04:47:17.232184 (+ 1237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:47:17.232795 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:47:17.232802 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:47:17.233165 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:47:17.233653 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:47:17.233667 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:47:17.234949 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:47:17.234952 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:47:17.235848 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:47:17.235857 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:47:17.236539 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:47:17.236542 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:47:17.236714 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:47:17.237372 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:47:17.237387 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:47:17.238979 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:47:17.240823 (+ 1844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:47:17.240828 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:47:17.240841 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:47:17.241208 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:47:17.241534 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:47:17.241537 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:47:17.241538 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:47:17.241763 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:47:17.242156 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:47:17.242166 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:47:17.242553 (+ 387us) client_negotiation.cc:769] Sending connection context
0925 04:47:17.242786 (+ 233us) client_negotiation.cc:240] Negotiation successful
0925 04:47:17.243194 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":165,"threads_started":1}
I20260925 04:47:17.245654 16126 heartbeater.cc:347] Connected to a master server at 127.0.98.126:33181
I20260925 04:47:17.245950 16126 heartbeater.cc:464] Registering TS with master...
I20260925 04:47:17.246668 16126 heartbeater.cc:511] Master 127.0.98.126:33181 requested a full tablet report, sending...
I20260925 04:47:17.247939 15801 ts_manager.cc:194] Registered new tserver with Master: c74e7ff9fd45472cbcc9cb1f76994e3e (127.0.98.66:43903)
I20260925 04:47:17.248579 15801 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.66@KRBTEST.COM'} at 127.0.98.66:49493
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/krb5kdc/kudu_127.0.98.67.keytab.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/krb5kdc/kudu_127.0.98.67.keytab.
I20260925 04:47:17.261298 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:0
--local_ip_for_outbound_sockets=127.0.98.67
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:33181
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:39863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:47:17.373853 16131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:47:17.374148 16131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:47:17.374222 16131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:47:17.378288 16131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:47:17.378386 16131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:47:17.378484 16131 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260925 04:47:17.383781 16131 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:39863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:33181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16131
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:47:17.384902 16131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:47:17.385949 16131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:47:17.395884 16139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:47:17.395877 16137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:47:17.395929 16131 server_base.cc:1034] running on GCE node
W20260925 04:47:17.395877 16136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:47:17.396658 16131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:47:17.397357 16131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:47:17.398556 16131 hybrid_clock.cc:648] HybridClock initialized: now 1790311637398512 us; error 50 us; skew 500 ppm
Sep 25 04:47:17 dist-test-slave-kghz krb5kdc[15756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311637, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:47:17.401564 16131 init.cc:377] Logged in from keytab as kudu/127.0.98.67@KRBTEST.COM (short username kudu)
I20260925 04:47:17.402640 16131 webserver.cc:460] Webserver started at http://127.0.98.67:42833/ using document root <none> and password file <none>
I20260925 04:47:17.403239 16131 fs_manager.cc:359] Metadata directory not provided
I20260925 04:47:17.403333 16131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:47:17.403587 16131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:47:17.405733 16131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-2/data/instance:
uuid: "0581358747614238835ea146d3d682ea"
format_stamp: "Formatted at 2026-09-25 04:47:17 on dist-test-slave-kghz"
server_key: "d398fe1da8d7289c549debda91682662"
server_key_iv: "288a1f4351930a9ac57305eb1f069db8"
server_key_version: "encryptionkey@0"
I20260925 04:47:17.406378 16131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-2/wal/instance:
uuid: "0581358747614238835ea146d3d682ea"
format_stamp: "Formatted at 2026-09-25 04:47:17 on dist-test-slave-kghz"
server_key: "d398fe1da8d7289c549debda91682662"
server_key_iv: "288a1f4351930a9ac57305eb1f069db8"
server_key_version: "encryptionkey@0"
I20260925 04:47:17.410228 16131 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260925 04:47:17.412847 16146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:47:17.414132 16131 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 04:47:17.414319 16131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-2/wal
uuid: "0581358747614238835ea146d3d682ea"
format_stamp: "Formatted at 2026-09-25 04:47:17 on dist-test-slave-kghz"
server_key: "d398fe1da8d7289c549debda91682662"
server_key_iv: "288a1f4351930a9ac57305eb1f069db8"
server_key_version: "encryptionkey@0"
I20260925 04:47:17.414448 16131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:47:17.425691 16131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:47:17.429318 16131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:47:17.429543 16131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:47:17.430176 16131 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:47:17.431092 16131 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:47:17.431165 16131 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:47:17.431234 16131 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:47:17.431285 16131 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:47:17.440829 16131 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:43709
I20260925 04:47:17.440853 16259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:43709 every 8 connection(s)
I20260925 04:47:17.441800 16131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
I20260925 04:47:17.447911 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 16131
I20260925 04:47:17.448019 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-2/wal/instance
I20260925 04:47:17.448298 393 external_mini_cluster.cc:1428] Setting key f9b2d43782fd02b67eb7c1f0bb420c48
Sep 25 04:47:17 dist-test-slave-kghz krb5kdc[15756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311637, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:47:17.455935 15840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:17.443729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:60747 (local address 127.0.98.126:33181)
0925 04:47:17.443906 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 04:47:17.443910 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:47:17.444981 (+ 1071us) server_negotiation.cc:407] Connection header received
0925 04:47:17.445965 (+ 984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:47:17.445968 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:47:17.446035 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:47:17.446119 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:47:17.447430 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:47:17.448071 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:47:17.449438 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:47:17.449681 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:47:17.449795 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:47:17.452423 (+ 2628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:47:17.452432 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:47:17.452434 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:47:17.452460 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:47:17.453802 (+ 1342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:47:17.454369 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:47:17.454373 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:47:17.454375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:47:17.454461 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:47:17.454821 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:47:17.454824 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:47:17.454826 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:47:17.455021 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:47:17.455128 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 04:47:17.455645 (+ 517us) server_negotiation.cc:299] Negotiation successful
0925 04:47:17.455771 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 04:47:17.456751 16262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:17.444147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:33181 (local address 127.0.98.67:60747)
0925 04:47:17.444821 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0925 04:47:17.444865 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 04:47:17.445739 (+ 874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:47:17.446376 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:47:17.446384 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:47:17.446771 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:47:17.447273 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:47:17.447287 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:47:17.448227 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:47:17.448230 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:47:17.449308 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:47:17.449319 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:47:17.449841 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:47:17.449845 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:47:17.450040 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:47:17.450648 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:47:17.450664 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:47:17.452252 (+ 1588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:47:17.453933 (+ 1681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:47:17.453939 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:47:17.453955 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:47:17.454227 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:47:17.454565 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:47:17.454569 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:47:17.454571 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:47:17.454715 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:47:17.455127 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:47:17.455135 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:47:17.455402 (+ 267us) client_negotiation.cc:769] Sending connection context
0925 04:47:17.455591 (+ 189us) client_negotiation.cc:240] Negotiation successful
0925 04:47:17.455926 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":116,"threads_started":1}
I20260925 04:47:17.457937 16260 heartbeater.cc:347] Connected to a master server at 127.0.98.126:33181
I20260925 04:47:17.458190 16260 heartbeater.cc:464] Registering TS with master...
I20260925 04:47:17.458817 16260 heartbeater.cc:511] Master 127.0.98.126:33181 requested a full tablet report, sending...
I20260925 04:47:17.459844 15801 ts_manager.cc:194] Registered new tserver with Master: 0581358747614238835ea146d3d682ea (127.0.98.67:43709)
I20260925 04:47:17.460395 15801 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.67@KRBTEST.COM'} at 127.0.98.67:60747
I20260925 04:47:17.462126 393 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 04:47:17.471076 15840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:17.464137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36844 (local address 127.0.98.126:33181)
0925 04:47:17.464257 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 04:47:17.464268 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 04:47:17.464444 (+ 176us) server_negotiation.cc:407] Connection header received
0925 04:47:17.464550 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:47:17.464552 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:47:17.464594 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:47:17.464659 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:47:17.465294 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:47:17.465812 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:47:17.466680 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:47:17.466890 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:47:17.466969 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:47:17.468206 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:47:17.468209 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:47:17.468211 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:47:17.468232 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:47:17.469470 (+ 1238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:47:17.469873 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:47:17.469879 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:47:17.469880 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:47:17.469966 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:47:17.470207 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:47:17.470209 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:47:17.470211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:47:17.470486 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:47:17.470627 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0925 04:47:17.470855 (+ 228us) server_negotiation.cc:299] Negotiation successful
0925 04:47:17.470955 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 04:47:17.473978 15801 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36844:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:47:17.477083 15801 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:47:17.492202 16272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:17.486061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59896 (local address 127.0.98.65:41669)
0925 04:47:17.487551 (+ 1490us) server_negotiation.cc:206] Beginning negotiation
0925 04:47:17.487567 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 04:47:17.487652 (+ 85us) server_negotiation.cc:407] Connection header received
0925 04:47:17.487855 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:47:17.487861 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:47:17.488049 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:47:17.488210 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:47:17.488446 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:47:17.489203 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:47:17.490732 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:47:17.491454 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:47:17.491567 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:47:17.491728 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 04:47:17.491936 (+ 208us) server_negotiation.cc:299] Negotiation successful
0925 04:47:17.492046 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1382,"thread_start_us":87,"threads_started":1}
I20260925 04:47:17.492209 16276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:17.485897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:41669 (local address 127.0.0.1:59896)
0925 04:47:17.487549 (+ 1652us) negotiation.cc:107] Waiting for socket to connect
0925 04:47:17.487565 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:47:17.487691 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:47:17.488204 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:47:17.488207 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:47:17.488223 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:47:17.488325 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:47:17.488334 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:47:17.489362 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:47:17.489365 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:47:17.490609 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:47:17.490618 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:47:17.491597 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:47:17.491601 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:47:17.491723 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:47:17.491767 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 04:47:17.491891 (+ 124us) client_negotiation.cc:240] Negotiation successful
0925 04:47:17.492023 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1597,"mutex_wait_us":666,"thread_start_us":90,"threads_started":1}
I20260925 04:47:17.492574 16271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:17.486248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:43903 (local address 127.0.0.1:57512)
0925 04:47:17.487010 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0925 04:47:17.487023 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:47:17.487297 (+ 274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:47:17.488074 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:47:17.488078 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:47:17.488103 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:47:17.488217 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:47:17.488225 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:47:17.489595 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:47:17.489597 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:47:17.490999 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:47:17.491010 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:47:17.492142 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:47:17.492145 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:47:17.492239 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:47:17.492294 (+ 55us) client_negotiation.cc:769] Sending connection context
0925 04:47:17.492371 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 04:47:17.492446 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":693,"mutex_wait_us":101,"thread_start_us":94,"threads_started":1}
I20260925 04:47:17.492986 16275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:17.486524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57512 (local address 127.0.98.66:43903)
0925 04:47:17.487297 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0925 04:47:17.487314 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 04:47:17.487338 (+ 24us) server_negotiation.cc:407] Connection header received
0925 04:47:17.487544 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:47:17.487549 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:47:17.487730 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:47:17.487895 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:47:17.488351 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:47:17.489466 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:47:17.491161 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:47:17.491864 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:47:17.491958 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:47:17.492085 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 04:47:17.492698 (+ 613us) server_negotiation.cc:299] Negotiation successful
0925 04:47:17.492816 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":131,"threads_started":1}
I20260925 04:47:17.493088 16273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:17.486165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59550 (local address 127.0.98.67:43709)
0925 04:47:17.486848 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0925 04:47:17.486860 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 04:47:17.487190 (+ 330us) server_negotiation.cc:407] Connection header received
0925 04:47:17.487504 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:47:17.487511 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:47:17.487729 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:47:17.487895 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:47:17.488189 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:47:17.489393 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:47:17.491127 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:47:17.491945 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:47:17.492046 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:47:17.492182 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 04:47:17.492820 (+ 638us) server_negotiation.cc:299] Negotiation successful
0925 04:47:17.492921 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":170,"threads_started":1}
I20260925 04:47:17.492725 16274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:17.485899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:43709 (local address 127.0.0.1:59550)
0925 04:47:17.486862 (+ 963us) negotiation.cc:107] Waiting for socket to connect
0925 04:47:17.486918 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 04:47:17.487306 (+ 388us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:47:17.487888 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:47:17.487894 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:47:17.487916 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:47:17.488057 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:47:17.488066 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:47:17.489554 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:47:17.489557 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:47:17.490999 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:47:17.491010 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:47:17.492252 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:47:17.492257 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:47:17.492416 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:47:17.492438 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 04:47:17.492507 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 04:47:17.492579 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":770,"mutex_wait_us":74,"thread_start_us":90,"threads_started":1}
I20260925 04:47:17.495841 15926 tablet_service.cc:1467] Processing CreateTablet for tablet 55ba235f8c6a42adab4691ad1c29dc60 (DEFAULT_TABLE table=test-table [id=336733467ffa4b7bbfb92e48d4dda0ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:47:17.496227 16194 tablet_service.cc:1467] Processing CreateTablet for tablet 55ba235f8c6a42adab4691ad1c29dc60 (DEFAULT_TABLE table=test-table [id=336733467ffa4b7bbfb92e48d4dda0ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:47:17.496233 16060 tablet_service.cc:1467] Processing CreateTablet for tablet 55ba235f8c6a42adab4691ad1c29dc60 (DEFAULT_TABLE table=test-table [id=336733467ffa4b7bbfb92e48d4dda0ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:47:17.496912 15926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55ba235f8c6a42adab4691ad1c29dc60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:47:17.497246 16060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55ba235f8c6a42adab4691ad1c29dc60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:47:17.497255 16194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55ba235f8c6a42adab4691ad1c29dc60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:47:17.504201 16277 tablet_bootstrap.cc:492] T 55ba235f8c6a42adab4691ad1c29dc60 P 7a05a5d169f8498a823f5f8b7ac15f92: Bootstrap starting.
I20260925 04:47:17.505303 16279 tablet_bootstrap.cc:492] T 55ba235f8c6a42adab4691ad1c29dc60 P c74e7ff9fd45472cbcc9cb1f76994e3e: Bootstrap starting.
I20260925 04:47:17.506448 16277 tablet_bootstrap.cc:654] T 55ba235f8c6a42adab4691ad1c29dc60 P 7a05a5d169f8498a823f5f8b7ac15f92: Neither blocks nor log segments found. Creating new log.
I20260925 04:47:17.506574 16278 tablet_bootstrap.cc:492] T 55ba235f8c6a42adab4691ad1c29dc60 P 0581358747614238835ea146d3d682ea: Bootstrap starting.
I20260925 04:47:17.507571 16277 log.cc:824] T 55ba235f8c6a42adab4691ad1c29dc60 P 7a05a5d169f8498a823f5f8b7ac15f92: Log is configured to *not* fsync() on all Append() calls
I20260925 04:47:17.507669 16279 tablet_bootstrap.cc:654] T 55ba235f8c6a42adab4691ad1c29dc60 P c74e7ff9fd45472cbcc9cb1f76994e3e: Neither blocks nor log segments found. Creating new log.
I20260925 04:47:17.508514 16279 log.cc:824] T 55ba235f8c6a42adab4691ad1c29dc60 P c74e7ff9fd45472cbcc9cb1f76994e3e: Log is configured to *not* fsync() on all Append() calls
I20260925 04:47:17.509116 16278 tablet_bootstrap.cc:654] T 55ba235f8c6a42adab4691ad1c29dc60 P 0581358747614238835ea146d3d682ea: Neither blocks nor log segments found. Creating new log.
I20260925 04:47:17.509862 16277 tablet_bootstrap.cc:492] T 55ba235f8c6a42adab4691ad1c29dc60 P 7a05a5d169f8498a823f5f8b7ac15f92: No bootstrap required, opened a new log
I20260925 04:47:17.509984 16278 log.cc:824] T 55ba235f8c6a42adab4691ad1c29dc60 P 0581358747614238835ea146d3d682ea: Log is configured to *not* fsync() on all Append() calls
I20260925 04:47:17.510090 16277 ts_tablet_manager.cc:1397] T 55ba235f8c6a42adab4691ad1c29dc60 P 7a05a5d169f8498a823f5f8b7ac15f92: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260925 04:47:17.510565 16279 tablet_bootstrap.cc:492] T 55ba235f8c6a42adab4691ad1c29dc60 P c74e7ff9fd45472cbcc9cb1f76994e3e: No bootstrap required, opened a new log
I20260925 04:47:17.510764 16279 ts_tablet_manager.cc:1397] T 55ba235f8c6a42adab4691ad1c29dc60 P c74e7ff9fd45472cbcc9cb1f76994e3e: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 04:47:17.511852 16278 tablet_bootstrap.cc:492] T 55ba235f8c6a42adab4691ad1c29dc60 P 0581358747614238835ea146d3d682ea: No bootstrap required, opened a new log
I20260925 04:47:17.512022 16278 ts_tablet_manager.cc:1397] T 55ba235f8c6a42adab4691ad1c29dc60 P 0581358747614238835ea146d3d682ea: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 04:47:17.513314 16277 raft_consensus.cc:348] T 55ba235f8c6a42adab4691ad1c29dc60 P 7a05a5d169f8498a823f5f8b7ac15f92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a05a5d169f8498a823f5f8b7ac15f92" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41669 } } peers { permanent_uuid: "0581358747614238835ea146d3d682ea" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43709 } } peers { permanent_uuid: "c74e7ff9fd45472cbcc9cb1f76994e3e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43903 } }
I20260925 04:47:17.513551 16277 raft_consensus.cc:374] T 55ba235f8c6a42adab4691ad1c29dc60 P 7a05a5d169f8498a823f5f8b7ac15f92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:47:17.513623 16277 raft_consensus.cc:729] T 55ba235f8c6a42adab4691ad1c29dc60 P 7a05a5d169f8498a823f5f8b7ac15f92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a05a5d169f8498a823f5f8b7ac15f92, State: Initialized, Role: FOLLOWER
I20260925 04:47:17.514087 16277 consensus_queue.cc:260] T 55ba235f8c6a42adab4691ad1c29dc60 P 7a05a5d169f8498a823f5f8b7ac15f92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a05a5d169f8498a823f5f8b7ac15f92" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41669 } } peers { permanent_uuid: "0581358747614238835ea146d3d682ea" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43709 } } peers { permanent_uuid: "c74e7ff9fd45472cbcc9cb1f76994e3e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43903 } }
I20260925 04:47:17.514729 15992 heartbeater.cc:503] Master 127.0.98.126:33181 was elected leader, sending a full tablet report...
I20260925 04:47:17.514554 16279 raft_consensus.cc:348] T 55ba235f8c6a42adab4691ad1c29dc60 P c74e7ff9fd45472cbcc9cb1f76994e3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a05a5d169f8498a823f5f8b7ac15f92" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41669 } } peers { permanent_uuid: "0581358747614238835ea146d3d682ea" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43709 } } peers { permanent_uuid: "c74e7ff9fd45472cbcc9cb1f76994e3e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43903 } }
I20260925 04:47:17.514820 16279 raft_consensus.cc:374] T 55ba235f8c6a42adab4691ad1c29dc60 P c74e7ff9fd45472cbcc9cb1f76994e3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:47:17.514869 16279 raft_consensus.cc:729] T 55ba235f8c6a42adab4691ad1c29dc60 P c74e7ff9fd45472cbcc9cb1f76994e3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c74e7ff9fd45472cbcc9cb1f76994e3e, State: Initialized, Role: FOLLOWER
I20260925 04:47:17.515091 16277 ts_tablet_manager.cc:1428] T 55ba235f8c6a42adab4691ad1c29dc60 P 7a05a5d169f8498a823f5f8b7ac15f92: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20260925 04:47:17.515317 16279 consensus_queue.cc:260] T 55ba235f8c6a42adab4691ad1c29dc60 P c74e7ff9fd45472cbcc9cb1f76994e3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a05a5d169f8498a823f5f8b7ac15f92" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41669 } } peers { permanent_uuid: "0581358747614238835ea146d3d682ea" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43709 } } peers { permanent_uuid: "c74e7ff9fd45472cbcc9cb1f76994e3e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43903 } }
I20260925 04:47:17.515789 16278 raft_consensus.cc:348] T 55ba235f8c6a42adab4691ad1c29dc60 P 0581358747614238835ea146d3d682ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a05a5d169f8498a823f5f8b7ac15f92" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41669 } } peers { permanent_uuid: "0581358747614238835ea146d3d682ea" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43709 } } peers { permanent_uuid: "c74e7ff9fd45472cbcc9cb1f76994e3e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43903 } }
I20260925 04:47:17.516084 16278 raft_consensus.cc:374] T 55ba235f8c6a42adab4691ad1c29dc60 P 0581358747614238835ea146d3d682ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:47:17.516140 16126 heartbeater.cc:503] Master 127.0.98.126:33181 was elected leader, sending a full tablet report...
I20260925 04:47:17.516171 16278 raft_consensus.cc:729] T 55ba235f8c6a42adab4691ad1c29dc60 P 0581358747614238835ea146d3d682ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0581358747614238835ea146d3d682ea, State: Initialized, Role: FOLLOWER
I20260925 04:47:17.516616 16278 consensus_queue.cc:260] T 55ba235f8c6a42adab4691ad1c29dc60 P 0581358747614238835ea146d3d682ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a05a5d169f8498a823f5f8b7ac15f92" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41669 } } peers { permanent_uuid: "0581358747614238835ea146d3d682ea" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43709 } } peers { permanent_uuid: "c74e7ff9fd45472cbcc9cb1f76994e3e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43903 } }
I20260925 04:47:17.517262 16260 heartbeater.cc:503] Master 127.0.98.126:33181 was elected leader, sending a full tablet report...
I20260925 04:47:17.517653 16278 ts_tablet_manager.cc:1428] T 55ba235f8c6a42adab4691ad1c29dc60 P 0581358747614238835ea146d3d682ea: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 04:47:17.518646 16284 raft_consensus.cc:482] T 55ba235f8c6a42adab4691ad1c29dc60 P c74e7ff9fd45472cbcc9cb1f76994e3e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:47:17.518785 16284 raft_consensus.cc:504] T 55ba235f8c6a42adab4691ad1c29dc60 P c74e7ff9fd45472cbcc9cb1f76994e3e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a05a5d169f8498a823f5f8b7ac15f92" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41669 } } peers { permanent_uuid: "0581358747614238835ea146d3d682ea" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43709 } } peers { permanent_uuid: "c74e7ff9fd45472cbcc9cb1f76994e3e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43903 } }
I20260925 04:47:17.518981 16279 ts_tablet_manager.cc:1428] T 55ba235f8c6a42adab4691ad1c29dc60 P c74e7ff9fd45472cbcc9cb1f76994e3e: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260925 04:47:17.519965 16284 leader_election.cc:288] T 55ba235f8c6a42adab4691ad1c29dc60 P c74e7ff9fd45472cbcc9cb1f76994e3e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a05a5d169f8498a823f5f8b7ac15f92 (127.0.98.65:41669), 0581358747614238835ea146d3d682ea (127.0.98.67:43709)
I20260925 04:47:17.524197 16272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:17.520505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:57567 (local address 127.0.98.65:41669)
0925 04:47:17.520689 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 04:47:17.520695 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:47:17.520714 (+ 19us) server_negotiation.cc:407] Connection header received
0925 04:47:17.520856 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:47:17.520860 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:47:17.520915 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:47:17.520963 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:47:17.521387 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:47:17.522014 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:47:17.523245 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:47:17.523651 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:47:17.523774 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:47:17.523836 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 04:47:17.524012 (+ 176us) server_negotiation.cc:299] Negotiation successful
0925 04:47:17.524078 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260925 04:47:17.524223 16287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:17.520150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:41669 (local address 127.0.98.66:57567)
0925 04:47:17.520631 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0925 04:47:17.520647 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:47:17.520731 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:47:17.521090 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:47:17.521093 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:47:17.521110 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:47:17.521285 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:47:17.521292 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:47:17.522107 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:47:17.522112 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:47:17.523114 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:47:17.523123 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:47:17.523777 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:47:17.523780 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:47:17.523879 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:47:17.523908 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 04:47:17.523996 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 04:47:17.524071 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"mutex_wait_us":166,"thread_start_us":99,"threads_started":1}
I20260925 04:47:17.524366 16273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:17.520532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:46733 (local address 127.0.98.67:43709)
0925 04:47:17.520745 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 04:47:17.520750 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:47:17.520764 (+ 14us) server_negotiation.cc:407] Connection header received
0925 04:47:17.520812 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:47:17.520818 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:47:17.520863 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:47:17.520936 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:47:17.521279 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:47:17.522032 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:47:17.523341 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:47:17.523796 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:47:17.523921 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:47:17.523981 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 04:47:17.524189 (+ 208us) server_negotiation.cc:299] Negotiation successful
0925 04:47:17.524258 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149}
I20260925 04:47:17.524366 16128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:17.520150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:43709 (local address 127.0.98.66:46733)
0925 04:47:17.520257 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0925 04:47:17.520272 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:47:17.520359 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:47:17.520975 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:47:17.520978 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:47:17.520997 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:47:17.521140 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:47:17.521148 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:47:17.522145 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:47:17.522148 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:47:17.523110 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:47:17.523118 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:47:17.523923 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:47:17.523926 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:47:17.524046 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:47:17.524086 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 04:47:17.524175 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 04:47:17.524246 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":62}
I20260925 04:47:17.524972 15946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55ba235f8c6a42adab4691ad1c29dc60" candidate_uuid: "c74e7ff9fd45472cbcc9cb1f76994e3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a05a5d169f8498a823f5f8b7ac15f92" is_pre_election: true
I20260925 04:47:17.524972 16214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55ba235f8c6a42adab4691ad1c29dc60" candidate_uuid: "c74e7ff9fd45472cbcc9cb1f76994e3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0581358747614238835ea146d3d682ea" is_pre_election: true
I20260925 04:47:17.525262 16214 raft_consensus.cc:2454] T 55ba235f8c6a42adab4691ad1c29dc60 P 0581358747614238835ea146d3d682ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c74e7ff9fd45472cbcc9cb1f76994e3e in term 0.
I20260925 04:47:17.525262 15946 raft_consensus.cc:2454] T 55ba235f8c6a42adab4691ad1c29dc60 P 7a05a5d169f8498a823f5f8b7ac15f92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c74e7ff9fd45472cbcc9cb1f76994e3e in term 0.
I20260925 04:47:17.525676 16016 leader_election.cc:302] T 55ba235f8c6a42adab4691ad1c29dc60 P c74e7ff9fd45472cbcc9cb1f76994e3e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0581358747614238835ea146d3d682ea, c74e7ff9fd45472cbcc9cb1f76994e3e; no voters:
I20260925 04:47:17.525978 16284 raft_consensus.cc:2781] T 55ba235f8c6a42adab4691ad1c29dc60 P c74e7ff9fd45472cbcc9cb1f76994e3e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:47:17.526059 16284 raft_consensus.cc:482] T 55ba235f8c6a42adab4691ad1c29dc60 P c74e7ff9fd45472cbcc9cb1f76994e3e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:47:17.526113 16284 raft_consensus.cc:3037] T 55ba235f8c6a42adab4691ad1c29dc60 P c74e7ff9fd45472cbcc9cb1f76994e3e [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:47:17.527333 16284 raft_consensus.cc:504] T 55ba235f8c6a42adab4691ad1c29dc60 P c74e7ff9fd45472cbcc9cb1f76994e3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a05a5d169f8498a823f5f8b7ac15f92" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41669 } } peers { permanent_uuid: "0581358747614238835ea146d3d682ea" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43709 } } peers { permanent_uuid: "c74e7ff9fd45472cbcc9cb1f76994e3e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43903 } }
I20260925 04:47:17.527735 16284 leader_election.cc:288] T 55ba235f8c6a42adab4691ad1c29dc60 P c74e7ff9fd45472cbcc9cb1f76994e3e [CANDIDATE]: Term 1 election: Requested vote from peers 7a05a5d169f8498a823f5f8b7ac15f92 (127.0.98.65:41669), 0581358747614238835ea146d3d682ea (127.0.98.67:43709)
I20260925 04:47:17.528052 16214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55ba235f8c6a42adab4691ad1c29dc60" candidate_uuid: "c74e7ff9fd45472cbcc9cb1f76994e3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0581358747614238835ea146d3d682ea"
I20260925 04:47:17.528044 15946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55ba235f8c6a42adab4691ad1c29dc60" candidate_uuid: "c74e7ff9fd45472cbcc9cb1f76994e3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a05a5d169f8498a823f5f8b7ac15f92"
I20260925 04:47:17.528170 16214 raft_consensus.cc:3037] T 55ba235f8c6a42adab4691ad1c29dc60 P 0581358747614238835ea146d3d682ea [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:47:17.528189 15946 raft_consensus.cc:3037] T 55ba235f8c6a42adab4691ad1c29dc60 P 7a05a5d169f8498a823f5f8b7ac15f92 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:47:17.529398 15946 raft_consensus.cc:2454] T 55ba235f8c6a42adab4691ad1c29dc60 P 7a05a5d169f8498a823f5f8b7ac15f92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c74e7ff9fd45472cbcc9cb1f76994e3e in term 1.
I20260925 04:47:17.529398 16214 raft_consensus.cc:2454] T 55ba235f8c6a42adab4691ad1c29dc60 P 0581358747614238835ea146d3d682ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c74e7ff9fd45472cbcc9cb1f76994e3e in term 1.
I20260925 04:47:17.529881 16014 leader_election.cc:302] T 55ba235f8c6a42adab4691ad1c29dc60 P c74e7ff9fd45472cbcc9cb1f76994e3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0581358747614238835ea146d3d682ea, 7a05a5d169f8498a823f5f8b7ac15f92, c74e7ff9fd45472cbcc9cb1f76994e3e; no voters:
I20260925 04:47:17.530069 16284 raft_consensus.cc:2781] T 55ba235f8c6a42adab4691ad1c29dc60 P c74e7ff9fd45472cbcc9cb1f76994e3e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:47:17.530195 16284 raft_consensus.cc:686] T 55ba235f8c6a42adab4691ad1c29dc60 P c74e7ff9fd45472cbcc9cb1f76994e3e [term 1 LEADER]: Becoming Leader. State: Replica: c74e7ff9fd45472cbcc9cb1f76994e3e, State: Running, Role: LEADER
I20260925 04:47:17.530579 16284 consensus_queue.cc:237] T 55ba235f8c6a42adab4691ad1c29dc60 P c74e7ff9fd45472cbcc9cb1f76994e3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a05a5d169f8498a823f5f8b7ac15f92" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41669 } } peers { permanent_uuid: "0581358747614238835ea146d3d682ea" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43709 } } peers { permanent_uuid: "c74e7ff9fd45472cbcc9cb1f76994e3e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43903 } }
I20260925 04:47:17.533806 15800 catalog_manager.cc:5495] T 55ba235f8c6a42adab4691ad1c29dc60 P c74e7ff9fd45472cbcc9cb1f76994e3e reported cstate change: term changed from 0 to 1, leader changed from <none> to c74e7ff9fd45472cbcc9cb1f76994e3e (127.0.98.66). New cstate: current_term: 1 leader_uuid: "c74e7ff9fd45472cbcc9cb1f76994e3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a05a5d169f8498a823f5f8b7ac15f92" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41669 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0581358747614238835ea146d3d682ea" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c74e7ff9fd45472cbcc9cb1f76994e3e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43903 } health_report { overall_health: HEALTHY } } }
I20260925 04:47:17.545332 16275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:17.542061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57518 (local address 127.0.98.66:43903)
0925 04:47:17.542184 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 04:47:17.542188 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:47:17.542201 (+ 13us) server_negotiation.cc:407] Connection header received
0925 04:47:17.542340 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:47:17.542342 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:47:17.542399 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:47:17.542490 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:47:17.542753 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:47:17.543234 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:47:17.544060 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:47:17.544262 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:47:17.544331 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:47:17.544595 (+ 264us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:47:17.544994 (+ 399us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:47:17.545083 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 04:47:17.545161 (+ 78us) server_negotiation.cc:299] Negotiation successful
0925 04:47:17.545207 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 04:47:17.551194 16214 raft_consensus.cc:1261] T 55ba235f8c6a42adab4691ad1c29dc60 P 0581358747614238835ea146d3d682ea [term 1 FOLLOWER]: Refusing update from remote peer c74e7ff9fd45472cbcc9cb1f76994e3e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:47:17.551195 15946 raft_consensus.cc:1261] T 55ba235f8c6a42adab4691ad1c29dc60 P 7a05a5d169f8498a823f5f8b7ac15f92 [term 1 FOLLOWER]: Refusing update from remote peer c74e7ff9fd45472cbcc9cb1f76994e3e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:47:17.551903 16285 consensus_queue.cc:1035] T 55ba235f8c6a42adab4691ad1c29dc60 P c74e7ff9fd45472cbcc9cb1f76994e3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a05a5d169f8498a823f5f8b7ac15f92" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:47:17.552124 16284 consensus_queue.cc:1035] T 55ba235f8c6a42adab4691ad1c29dc60 P c74e7ff9fd45472cbcc9cb1f76994e3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0581358747614238835ea146d3d682ea" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:47:17.562947 16291 mvcc.cc:205] Tried to move back new op lower bound from 7333116467401928704 to 7333116467327721472. Current Snapshot: MvccSnapshot[applied={T|T < 7333116467401928704}]
I20260925 04:47:17.567966 15801 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36844:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:47:17.568148 15801 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36844:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:47:17.570489 15801 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a175a4ff2ea44c5ca7a8f69fec22566a: Sending DeleteTablet for 3 replicas of tablet 55ba235f8c6a42adab4691ad1c29dc60
I20260925 04:47:17.571345 15926 tablet_service.cc:1514] Processing DeleteTablet for tablet 55ba235f8c6a42adab4691ad1c29dc60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:47:17 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:59896
I20260925 04:47:17.571485 16060 tablet_service.cc:1514] Processing DeleteTablet for tablet 55ba235f8c6a42adab4691ad1c29dc60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:47:17 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:57512
I20260925 04:47:17.571487 16194 tablet_service.cc:1514] Processing DeleteTablet for tablet 55ba235f8c6a42adab4691ad1c29dc60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:47:17 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:59550
I20260925 04:47:17.572125 16301 tablet_replica.cc:330] T 55ba235f8c6a42adab4691ad1c29dc60 P 0581358747614238835ea146d3d682ea: stopping tablet replica
I20260925 04:47:17.572288 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 15863
I20260925 04:47:17.572522 16301 raft_consensus.cc:2229] T 55ba235f8c6a42adab4691ad1c29dc60 P 0581358747614238835ea146d3d682ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:47:17.573999 16301 raft_consensus.cc:2258] T 55ba235f8c6a42adab4691ad1c29dc60 P 0581358747614238835ea146d3d682ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:47:17.574672 16300 tablet_replica.cc:330] T 55ba235f8c6a42adab4691ad1c29dc60 P c74e7ff9fd45472cbcc9cb1f76994e3e: stopping tablet replica
I20260925 04:47:17.575026 16300 raft_consensus.cc:2229] T 55ba235f8c6a42adab4691ad1c29dc60 P c74e7ff9fd45472cbcc9cb1f76994e3e [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:47:17.575438 16300 raft_consensus.cc:2258] T 55ba235f8c6a42adab4691ad1c29dc60 P c74e7ff9fd45472cbcc9cb1f76994e3e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:47:17.575927 16301 ts_tablet_manager.cc:1905] T 55ba235f8c6a42adab4691ad1c29dc60 P 0581358747614238835ea146d3d682ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:47:17.576815 16300 ts_tablet_manager.cc:1905] T 55ba235f8c6a42adab4691ad1c29dc60 P c74e7ff9fd45472cbcc9cb1f76994e3e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:47:17.578842 16301 ts_tablet_manager.cc:1918] T 55ba235f8c6a42adab4691ad1c29dc60 P 0581358747614238835ea146d3d682ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:47:17.579023 16301 log.cc:1196] T 55ba235f8c6a42adab4691ad1c29dc60 P 0581358747614238835ea146d3d682ea: Deleting WAL directory at /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-2/wal/wals/55ba235f8c6a42adab4691ad1c29dc60
I20260925 04:47:17.579445 16301 ts_tablet_manager.cc:1939] T 55ba235f8c6a42adab4691ad1c29dc60 P 0581358747614238835ea146d3d682ea: Deleting consensus metadata
I20260925 04:47:17.579658 16300 ts_tablet_manager.cc:1918] T 55ba235f8c6a42adab4691ad1c29dc60 P c74e7ff9fd45472cbcc9cb1f76994e3e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:47:17.579823 16300 log.cc:1196] T 55ba235f8c6a42adab4691ad1c29dc60 P c74e7ff9fd45472cbcc9cb1f76994e3e: Deleting WAL directory at /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-1/wal/wals/55ba235f8c6a42adab4691ad1c29dc60
I20260925 04:47:17.580211 16300 ts_tablet_manager.cc:1939] T 55ba235f8c6a42adab4691ad1c29dc60 P c74e7ff9fd45472cbcc9cb1f76994e3e: Deleting consensus metadata
W20260925 04:47:17.580806 15786 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.65:41669 (error 108)
I20260925 04:47:17.580981 15788 catalog_manager.cc:4841] TS 0581358747614238835ea146d3d682ea (127.0.98.67:43709): tablet 55ba235f8c6a42adab4691ad1c29dc60 (table test-table [id=336733467ffa4b7bbfb92e48d4dda0ab]) successfully deleted
I20260925 04:47:17.581314 15785 catalog_manager.cc:4841] TS c74e7ff9fd45472cbcc9cb1f76994e3e (127.0.98.66:43903): tablet 55ba235f8c6a42adab4691ad1c29dc60 (table test-table [id=336733467ffa4b7bbfb92e48d4dda0ab]) successfully deleted
I20260925 04:47:17.582033 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 15997
I20260925 04:47:17.582412 16274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:17.582058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:41669 (local address 127.0.0.1:59900)
0925 04:47:17.582221 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 04:47:17.582313 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.65:41669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260925 04:47:17.583645 15786 catalog_manager.cc:4584] TS 7a05a5d169f8498a823f5f8b7ac15f92 (127.0.98.65:41669): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 55ba235f8c6a42adab4691ad1c29dc60: Network error: Client connection negotiation failed: client connection to 127.0.98.65:41669: connect: Connection refused (error 111)
I20260925 04:47:17.588403 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 16131
I20260925 04:47:17.594153 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 15769
I20260925 04:47:37.600620 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:33181
--webserver_interface=127.0.98.126
--webserver_port=38533
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:39863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:33181
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:47:37.711119 16303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:47:37.711391 16303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:47:37.711433 16303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:47:37.715603 16303 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 04:47:37.715665 16303 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 04:47:37.715682 16303 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 04:47:37.715695 16303 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:47:37.715714 16303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:47:37.715732 16303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:47:37.715749 16303 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:47:37.715763 16303 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:47:37.721077 16303 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:39863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:33181
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:33181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=38533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:47:37.722082 16303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:47:37.723153 16303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 04:47:37.736495 16303 server_base.cc:1034] running on GCE node
W20260925 04:47:37.736529 16309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:47:37.736649 16310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:47:37.736538 16312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:47:37.737282 16303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:47:37.738555 16303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:47:37.739712 16303 hybrid_clock.cc:648] HybridClock initialized: now 1790311657739695 us; error 32 us; skew 500 ppm
Sep 25 04:47:37 dist-test-slave-kghz krb5kdc[15756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311657, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:47:37.742556 16303 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260925 04:47:37.743522 16303 webserver.cc:460] Webserver started at http://127.0.98.126:38533/ using document root <none> and password file <none>
I20260925 04:47:37.744127 16303 fs_manager.cc:359] Metadata directory not provided
I20260925 04:47:37.744201 16303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:47:37.748081 16303 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260925 04:47:37.750366 16319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:47:37.751713 16303 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:47:37.751870 16303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/master-0/wal
uuid: "a175a4ff2ea44c5ca7a8f69fec22566a"
format_stamp: "Formatted at 2026-09-25 04:47:16 on dist-test-slave-kghz"
server_key: "184464def56b690e2a70b606620f3760"
server_key_iv: "b9c5f2a8beaab36785dae5e8815eeff3"
server_key_version: "encryptionkey@0"
I20260925 04:47:37.752470 16303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:47:37.784802 16303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:47:37.788383 16303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:47:37.788645 16303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:47:37.796036 16303 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:33181
I20260925 04:47:37.796339 16371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:33181 every 8 connection(s)
I20260925 04:47:37.797170 16303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
I20260925 04:47:37.797498 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 16303
I20260925 04:47:37.797951 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:41669
--local_ip_for_outbound_sockets=127.0.98.65
--tserver_master_addrs=127.0.98.126:33181
--webserver_port=38693
--webserver_interface=127.0.98.65
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:39863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 04:47:37.801723 16372 sys_catalog.cc:263] Verifying existing consensus state
I20260925 04:47:37.803503 16372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a175a4ff2ea44c5ca7a8f69fec22566a: Bootstrap starting.
I20260925 04:47:37.820535 16372 log.cc:824] T 00000000000000000000000000000000 P a175a4ff2ea44c5ca7a8f69fec22566a: Log is configured to *not* fsync() on all Append() calls
I20260925 04:47:37.829083 16372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a175a4ff2ea44c5ca7a8f69fec22566a: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 04:47:37.829481 16372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a175a4ff2ea44c5ca7a8f69fec22566a: Bootstrap complete.
I20260925 04:47:37.833597 16372 raft_consensus.cc:348] T 00000000000000000000000000000000 P a175a4ff2ea44c5ca7a8f69fec22566a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a175a4ff2ea44c5ca7a8f69fec22566a" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33181 } }
I20260925 04:47:37.834045 16372 raft_consensus.cc:729] T 00000000000000000000000000000000 P a175a4ff2ea44c5ca7a8f69fec22566a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a175a4ff2ea44c5ca7a8f69fec22566a, State: Initialized, Role: FOLLOWER
I20260925 04:47:37.834595 16372 consensus_queue.cc:260] T 00000000000000000000000000000000 P a175a4ff2ea44c5ca7a8f69fec22566a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a175a4ff2ea44c5ca7a8f69fec22566a" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33181 } }
I20260925 04:47:37.834749 16372 raft_consensus.cc:388] T 00000000000000000000000000000000 P a175a4ff2ea44c5ca7a8f69fec22566a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:47:37.834846 16372 raft_consensus.cc:482] T 00000000000000000000000000000000 P a175a4ff2ea44c5ca7a8f69fec22566a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:47:37.834962 16372 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a175a4ff2ea44c5ca7a8f69fec22566a [term 1 FOLLOWER]: Advancing to term 2
I20260925 04:47:37.836902 16372 raft_consensus.cc:504] T 00000000000000000000000000000000 P a175a4ff2ea44c5ca7a8f69fec22566a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a175a4ff2ea44c5ca7a8f69fec22566a" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33181 } }
I20260925 04:47:37.837253 16372 leader_election.cc:302] T 00000000000000000000000000000000 P a175a4ff2ea44c5ca7a8f69fec22566a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a175a4ff2ea44c5ca7a8f69fec22566a; no voters:
I20260925 04:47:37.837594 16372 leader_election.cc:288] T 00000000000000000000000000000000 P a175a4ff2ea44c5ca7a8f69fec22566a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 04:47:37.837728 16376 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a175a4ff2ea44c5ca7a8f69fec22566a [term 2 FOLLOWER]: Leader election won for term 2
I20260925 04:47:37.837982 16376 raft_consensus.cc:686] T 00000000000000000000000000000000 P a175a4ff2ea44c5ca7a8f69fec22566a [term 2 LEADER]: Becoming Leader. State: Replica: a175a4ff2ea44c5ca7a8f69fec22566a, State: Running, Role: LEADER
I20260925 04:47:37.838379 16376 consensus_queue.cc:237] T 00000000000000000000000000000000 P a175a4ff2ea44c5ca7a8f69fec22566a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a175a4ff2ea44c5ca7a8f69fec22566a" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33181 } }
I20260925 04:47:37.838843 16372 sys_catalog.cc:564] T 00000000000000000000000000000000 P a175a4ff2ea44c5ca7a8f69fec22566a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:47:37.839789 16377 sys_catalog.cc:455] T 00000000000000000000000000000000 P a175a4ff2ea44c5ca7a8f69fec22566a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a175a4ff2ea44c5ca7a8f69fec22566a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a175a4ff2ea44c5ca7a8f69fec22566a" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33181 } } }
I20260925 04:47:37.839926 16377 sys_catalog.cc:458] T 00000000000000000000000000000000 P a175a4ff2ea44c5ca7a8f69fec22566a [sys.catalog]: This master's current role is: LEADER
I20260925 04:47:37.840283 16378 sys_catalog.cc:455] T 00000000000000000000000000000000 P a175a4ff2ea44c5ca7a8f69fec22566a [sys.catalog]: SysCatalogTable state changed. Reason: New leader a175a4ff2ea44c5ca7a8f69fec22566a. Latest consensus state: current_term: 2 leader_uuid: "a175a4ff2ea44c5ca7a8f69fec22566a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a175a4ff2ea44c5ca7a8f69fec22566a" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33181 } } }
I20260925 04:47:37.840425 16378 sys_catalog.cc:458] T 00000000000000000000000000000000 P a175a4ff2ea44c5ca7a8f69fec22566a [sys.catalog]: This master's current role is: LEADER
I20260925 04:47:37.840776 16385 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:47:37.845849 16385 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:47:37.846452 16385 catalog_manager.cc:1259] Loaded cluster ID: c272a46e210846a7a5d01e8e195412a3
I20260925 04:47:37.846509 16385 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:47:37.848871 16385 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:47:37.849521 16385 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a175a4ff2ea44c5ca7a8f69fec22566a: Loaded TSK: 0
I20260925 04:47:37.871759 16385 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a175a4ff2ea44c5ca7a8f69fec22566a: Generated new TSK 1
I20260925 04:47:37.872160 16385 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 04:47:37.920292 16374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:47:37.920568 16374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:47:37.920612 16374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:47:37.924744 16374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:47:37.924847 16374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:47:37.924938 16374 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260925 04:47:37.930429 16374 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:39863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:41669
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=38693
--tserver_master_addrs=127.0.98.126:33181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:47:37.931496 16374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:47:37.932680 16374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:47:37.942911 16400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:47:37.942929 16402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:47:37.943132 16399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:47:37.943351 16374 server_base.cc:1034] running on GCE node
I20260925 04:47:37.943806 16374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:47:37.944530 16374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:47:37.945730 16374 hybrid_clock.cc:648] HybridClock initialized: now 1790311657945714 us; error 39 us; skew 500 ppm
Sep 25 04:47:37 dist-test-slave-kghz krb5kdc[15756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311657, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:47:37.949134 16374 init.cc:377] Logged in from keytab as kudu/127.0.98.65@KRBTEST.COM (short username kudu)
I20260925 04:47:37.950232 16374 webserver.cc:460] Webserver started at http://127.0.98.65:38693/ using document root <none> and password file <none>
I20260925 04:47:37.950928 16374 fs_manager.cc:359] Metadata directory not provided
I20260925 04:47:37.951011 16374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:47:37.955538 16374 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260925 04:47:37.958105 16409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:47:37.959558 16374 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:47:37.959718 16374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-0/wal
uuid: "7a05a5d169f8498a823f5f8b7ac15f92"
format_stamp: "Formatted at 2026-09-25 04:47:16 on dist-test-slave-kghz"
server_key: "42d5f01b4c6f1a6ccbf7cd191e555c7d"
server_key_iv: "b2d33c06103ea5639fe80b0a86c5cfd6"
server_key_version: "encryptionkey@0"
I20260925 04:47:37.960327 16374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:47:37.976037 16374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:47:37.979504 16374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:47:37.979732 16374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:47:37.980432 16374 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:47:37.981679 16416 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 04:47:37.986102 16374 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 04:47:37.986171 16374 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260925 04:47:37.986222 16374 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 04:47:37.988103 16374 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 04:47:37.988165 16374 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260925 04:47:37.988431 16416 tablet_bootstrap.cc:492] T 55ba235f8c6a42adab4691ad1c29dc60 P 7a05a5d169f8498a823f5f8b7ac15f92: Bootstrap starting.
I20260925 04:47:37.999981 16523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:41669 every 8 connection(s)
I20260925 04:47:37.999982 16374 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:41669
I20260925 04:47:38.001372 16374 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
I20260925 04:47:38.006385 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 16374
I20260925 04:47:38.006842 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:43903
--local_ip_for_outbound_sockets=127.0.98.66
--tserver_master_addrs=127.0.98.126:33181
--webserver_port=33783
--webserver_interface=127.0.98.66
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:39863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 04:47:38.011330 16416 log.cc:824] T 55ba235f8c6a42adab4691ad1c29dc60 P 7a05a5d169f8498a823f5f8b7ac15f92: Log is configured to *not* fsync() on all Append() calls
Sep 25 04:47:38 dist-test-slave-kghz krb5kdc[15756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311657, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:47:38.020521 16416 tablet_bootstrap.cc:492] T 55ba235f8c6a42adab4691ad1c29dc60 P 7a05a5d169f8498a823f5f8b7ac15f92: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 04:47:38.021034 16416 tablet_bootstrap.cc:492] T 55ba235f8c6a42adab4691ad1c29dc60 P 7a05a5d169f8498a823f5f8b7ac15f92: Bootstrap complete.
I20260925 04:47:38.022130 16527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:38.003730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:33181 (local address 127.0.98.65:39955)
0925 04:47:38.004293 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0925 04:47:38.004336 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 04:47:38.005167 (+ 831us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:47:38.006377 (+ 1210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:47:38.006387 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:47:38.006896 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:47:38.007733 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:47:38.007761 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:47:38.009800 (+ 2039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:47:38.009804 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:47:38.011263 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:47:38.011278 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:47:38.012468 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:47:38.012473 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:47:38.012691 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:47:38.013469 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:47:38.013493 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:47:38.015614 (+ 2121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:47:38.018512 (+ 2898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:47:38.018518 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:47:38.018532 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:47:38.018893 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:47:38.019361 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:47:38.019364 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:47:38.019366 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:47:38.019535 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:47:38.020301 (+ 766us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:47:38.020309 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:47:38.020600 (+ 291us) client_negotiation.cc:769] Sending connection context
0925 04:47:38.020880 (+ 280us) client_negotiation.cc:240] Negotiation successful
0925 04:47:38.021202 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":150,"threads_started":1}
I20260925 04:47:38.022313 16526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:38.003668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:39955 (local address 127.0.98.126:33181)
0925 04:47:38.004047 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0925 04:47:38.004069 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 04:47:38.004467 (+ 398us) server_negotiation.cc:407] Connection header received
0925 04:47:38.005582 (+ 1115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:47:38.005592 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:47:38.005886 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:47:38.006231 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:47:38.007972 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:47:38.009620 (+ 1648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:47:38.011419 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:47:38.012267 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:47:38.012469 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:47:38.015863 (+ 3394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:47:38.015870 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:47:38.015887 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:47:38.015924 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:47:38.018376 (+ 2452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:47:38.019073 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:47:38.019080 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:47:38.019086 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:47:38.019252 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:47:38.019656 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:47:38.019659 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:47:38.019661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:47:38.020172 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:47:38.020323 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 04:47:38.020983 (+ 660us) server_negotiation.cc:299] Negotiation successful
0925 04:47:38.021337 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":118,"threads_started":1}
I20260925 04:47:38.022627 16416 ts_tablet_manager.cc:1397] T 55ba235f8c6a42adab4691ad1c29dc60 P 7a05a5d169f8498a823f5f8b7ac15f92: Time spent bootstrapping tablet: real 0.034s user 0.022s sys 0.011s
I20260925 04:47:38.025175 16524 heartbeater.cc:347] Connected to a master server at 127.0.98.126:33181
I20260925 04:47:38.025422 16524 heartbeater.cc:464] Registering TS with master...
I20260925 04:47:38.025950 16524 heartbeater.cc:511] Master 127.0.98.126:33181 requested a full tablet report, sending...
I20260925 04:47:38.025712 16416 raft_consensus.cc:348] T 55ba235f8c6a42adab4691ad1c29dc60 P 7a05a5d169f8498a823f5f8b7ac15f92 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a05a5d169f8498a823f5f8b7ac15f92" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41669 } } peers { permanent_uuid: "0581358747614238835ea146d3d682ea" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43709 } } peers { permanent_uuid: "c74e7ff9fd45472cbcc9cb1f76994e3e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43903 } }
I20260925 04:47:38.026202 16416 raft_consensus.cc:729] T 55ba235f8c6a42adab4691ad1c29dc60 P 7a05a5d169f8498a823f5f8b7ac15f92 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a05a5d169f8498a823f5f8b7ac15f92, State: Initialized, Role: FOLLOWER
I20260925 04:47:38.026736 16416 consensus_queue.cc:260] T 55ba235f8c6a42adab4691ad1c29dc60 P 7a05a5d169f8498a823f5f8b7ac15f92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a05a5d169f8498a823f5f8b7ac15f92" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41669 } } peers { permanent_uuid: "0581358747614238835ea146d3d682ea" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43709 } } peers { permanent_uuid: "c74e7ff9fd45472cbcc9cb1f76994e3e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43903 } }
I20260925 04:47:38.028796 16416 ts_tablet_manager.cc:1428] T 55ba235f8c6a42adab4691ad1c29dc60 P 7a05a5d169f8498a823f5f8b7ac15f92: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260925 04:47:38.029332 16336 ts_manager.cc:194] Registered new tserver with Master: 7a05a5d169f8498a823f5f8b7ac15f92 (127.0.98.65:41669)
I20260925 04:47:38.033299 16336 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.65@KRBTEST.COM'} at 127.0.98.65:39955
I20260925 04:47:38.036399 16524 heartbeater.cc:503] Master 127.0.98.126:33181 was elected leader, sending a full tablet report...
Sep 25 04:47:38 dist-test-slave-kghz krb5kdc[15756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311657, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for kudu/127.0.98.65@KRBTEST.COM
I20260925 04:47:38.043913 16532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:38.032410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34134 (local address 127.0.98.65:41669)
0925 04:47:38.033089 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0925 04:47:38.033100 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 04:47:38.033123 (+ 23us) server_negotiation.cc:407] Connection header received
0925 04:47:38.033190 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:47:38.033195 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:47:38.033381 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:47:38.033510 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:47:38.034410 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:47:38.035598 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:47:38.036846 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:47:38.037286 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:47:38.037417 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:47:38.040180 (+ 2763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:47:38.040190 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:47:38.040202 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:47:38.040227 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:47:38.041873 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:47:38.042478 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:47:38.042483 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:47:38.042484 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:47:38.042567 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:47:38.042916 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:47:38.042919 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:47:38.042920 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:47:38.043172 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:47:38.043272 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 04:47:38.043658 (+ 386us) server_negotiation.cc:299] Negotiation successful
0925 04:47:38.043792 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":117,"threads_started":1}
I20260925 04:47:38.043937 16531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:38.032326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:41669 (local address 127.0.0.1:34134)
0925 04:47:38.032779 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0925 04:47:38.032829 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 04:47:38.032962 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:47:38.033580 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:47:38.033588 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:47:38.034034 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:47:38.034200 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:47:38.034210 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:47:38.035750 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:47:38.035754 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:47:38.036723 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:47:38.036737 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:47:38.037416 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:47:38.037419 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:47:38.037559 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:47:38.038384 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:47:38.038397 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:47:38.040020 (+ 1623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:47:38.041993 (+ 1973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:47:38.042004 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:47:38.042009 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:47:38.042367 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:47:38.042682 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:47:38.042687 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:47:38.042691 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:47:38.042806 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:47:38.043286 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:47:38.043292 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:47:38.043417 (+ 125us) client_negotiation.cc:769] Sending connection context
0925 04:47:38.043556 (+ 139us) client_negotiation.cc:240] Negotiation successful
0925 04:47:38.043763 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":124,"threads_started":1}
I20260925 04:47:38.044952 16457 tablet_service.cc:1514] Processing DeleteTablet for tablet 55ba235f8c6a42adab4691ad1c29dc60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:47:17 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:34134
I20260925 04:47:38.046016 16533 tablet_replica.cc:330] stopping tablet replica
I20260925 04:47:38.046397 16533 raft_consensus.cc:2229] T 55ba235f8c6a42adab4691ad1c29dc60 P 7a05a5d169f8498a823f5f8b7ac15f92 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:47:38.046696 16533 raft_consensus.cc:2258] T 55ba235f8c6a42adab4691ad1c29dc60 P 7a05a5d169f8498a823f5f8b7ac15f92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:47:38.047683 16533 ts_tablet_manager.cc:1905] T 55ba235f8c6a42adab4691ad1c29dc60 P 7a05a5d169f8498a823f5f8b7ac15f92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:47:38.051458 16533 ts_tablet_manager.cc:1918] T 55ba235f8c6a42adab4691ad1c29dc60 P 7a05a5d169f8498a823f5f8b7ac15f92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:47:38.051584 16533 log.cc:1196] T 55ba235f8c6a42adab4691ad1c29dc60 P 7a05a5d169f8498a823f5f8b7ac15f92: Deleting WAL directory at /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-0/wal/wals/55ba235f8c6a42adab4691ad1c29dc60
I20260925 04:47:38.051875 16533 ts_tablet_manager.cc:1939] T 55ba235f8c6a42adab4691ad1c29dc60 P 7a05a5d169f8498a823f5f8b7ac15f92: Deleting consensus metadata
I20260925 04:47:38.053670 16321 catalog_manager.cc:4841] TS 7a05a5d169f8498a823f5f8b7ac15f92 (127.0.98.65:41669): tablet 55ba235f8c6a42adab4691ad1c29dc60 (table test-table [id=336733467ffa4b7bbfb92e48d4dda0ab]) successfully deleted
W20260925 04:47:38.124953 16528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:47:38.125227 16528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:47:38.125272 16528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:47:38.129267 16528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:47:38.129323 16528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:47:38.129381 16528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260925 04:47:38.134495 16528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:39863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:43903
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=33783
--tserver_master_addrs=127.0.98.126:33181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:47:38.135494 16528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:47:38.136461 16528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:47:38.146399 16541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:47:38.146417 16539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:47:38.146474 16528 server_base.cc:1034] running on GCE node
W20260925 04:47:38.146399 16538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:47:38.146999 16528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:47:38.147670 16528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:47:38.148829 16528 hybrid_clock.cc:648] HybridClock initialized: now 1790311658148814 us; error 26 us; skew 500 ppm
Sep 25 04:47:38 dist-test-slave-kghz krb5kdc[15756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311658, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:47:38.151641 16528 init.cc:377] Logged in from keytab as kudu/127.0.98.66@KRBTEST.COM (short username kudu)
I20260925 04:47:38.152405 16528 webserver.cc:460] Webserver started at http://127.0.98.66:33783/ using document root <none> and password file <none>
I20260925 04:47:38.152907 16528 fs_manager.cc:359] Metadata directory not provided
I20260925 04:47:38.152974 16528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:47:38.156912 16528 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 04:47:38.159082 16548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:47:38.160342 16528 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 04:47:38.160473 16528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-1/wal
uuid: "c74e7ff9fd45472cbcc9cb1f76994e3e"
format_stamp: "Formatted at 2026-09-25 04:47:17 on dist-test-slave-kghz"
server_key: "f957305e70af52a37fc92da714a64437"
server_key_iv: "9502451580e9c1b30f9ee1a05a402086"
server_key_version: "encryptionkey@0"
I20260925 04:47:38.161082 16528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:47:38.175061 16528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:47:38.178730 16528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:47:38.178957 16528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:47:38.179592 16528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:47:38.180454 16528 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:47:38.180507 16528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:47:38.180557 16528 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:47:38.180619 16528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:47:38.190064 16528 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:43903
I20260925 04:47:38.190100 16661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:43903 every 8 connection(s)
I20260925 04:47:38.191140 16528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
I20260925 04:47:38.193830 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 16528
I20260925 04:47:38.194310 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:43709
--local_ip_for_outbound_sockets=127.0.98.67
--tserver_master_addrs=127.0.98.126:33181
--webserver_port=42833
--webserver_interface=127.0.98.67
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:39863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 04:47:38 dist-test-slave-kghz krb5kdc[15756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311658, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:47:38.206862 16526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:38.193030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:54251 (local address 127.0.98.126:33181)
0925 04:47:38.193208 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 04:47:38.193212 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:47:38.194320 (+ 1108us) server_negotiation.cc:407] Connection header received
0925 04:47:38.195281 (+ 961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:47:38.195287 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:47:38.195354 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:47:38.195465 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:47:38.197150 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:47:38.197699 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:47:38.199607 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:47:38.199835 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:47:38.199918 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:47:38.203046 (+ 3128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:47:38.203049 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:47:38.203051 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:47:38.203078 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:47:38.204583 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:47:38.205219 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:47:38.205221 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:47:38.205223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:47:38.205295 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:47:38.205684 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:47:38.205686 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:47:38.205688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:47:38.205852 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:47:38.205922 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 04:47:38.206621 (+ 699us) server_negotiation.cc:299] Negotiation successful
0925 04:47:38.206741 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 04:47:38.207764 16664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:38.193396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:33181 (local address 127.0.98.66:54251)
0925 04:47:38.194134 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0925 04:47:38.194192 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 04:47:38.195064 (+ 872us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:47:38.195760 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:47:38.195773 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:47:38.196296 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:47:38.196981 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:47:38.196998 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:47:38.197841 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:47:38.197848 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:47:38.199468 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:47:38.199482 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:47:38.200004 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:47:38.200010 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:47:38.200259 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:47:38.200901 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:47:38.200927 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:47:38.202848 (+ 1921us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:47:38.204699 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:47:38.204708 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:47:38.204728 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:47:38.205109 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:47:38.205385 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:47:38.205391 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:47:38.205395 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:47:38.205585 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:47:38.205949 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:47:38.205955 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:47:38.206277 (+ 322us) client_negotiation.cc:769] Sending connection context
0925 04:47:38.206473 (+ 196us) client_negotiation.cc:240] Negotiation successful
0925 04:47:38.206790 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":155,"threads_started":1}
I20260925 04:47:38.208946 16662 heartbeater.cc:347] Connected to a master server at 127.0.98.126:33181
I20260925 04:47:38.209200 16662 heartbeater.cc:464] Registering TS with master...
I20260925 04:47:38.209693 16662 heartbeater.cc:511] Master 127.0.98.126:33181 requested a full tablet report, sending...
I20260925 04:47:38.210731 16336 ts_manager.cc:194] Registered new tserver with Master: c74e7ff9fd45472cbcc9cb1f76994e3e (127.0.98.66:43903)
I20260925 04:47:38.211301 16336 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.66@KRBTEST.COM'} at 127.0.98.66:54251
W20260925 04:47:38.307384 16665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:47:38.307715 16665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:47:38.307790 16665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:47:38.312140 16665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:47:38.312233 16665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:47:38.312325 16665 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260925 04:47:38.317767 16665 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:39863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:43709
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=42833
--tserver_master_addrs=127.0.98.126:33181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16665
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:47:38.318918 16665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:47:38.319952 16665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:47:38.329514 16673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:47:38.329511 16671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:47:38.329577 16665 server_base.cc:1034] running on GCE node
W20260925 04:47:38.329511 16670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:47:38.330299 16665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:47:38.330958 16665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:47:38.332151 16665 hybrid_clock.cc:648] HybridClock initialized: now 1790311658332132 us; error 32 us; skew 500 ppm
Sep 25 04:47:38 dist-test-slave-kghz krb5kdc[15756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311658, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:47:38.334976 16665 init.cc:377] Logged in from keytab as kudu/127.0.98.67@KRBTEST.COM (short username kudu)
I20260925 04:47:38.335784 16665 webserver.cc:460] Webserver started at http://127.0.98.67:42833/ using document root <none> and password file <none>
I20260925 04:47:38.336295 16665 fs_manager.cc:359] Metadata directory not provided
I20260925 04:47:38.336361 16665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:47:38.340771 16665 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 04:47:38.343027 16680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:47:38.344324 16665 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 04:47:38.344462 16665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-2/wal
uuid: "0581358747614238835ea146d3d682ea"
format_stamp: "Formatted at 2026-09-25 04:47:17 on dist-test-slave-kghz"
server_key: "d398fe1da8d7289c549debda91682662"
server_key_iv: "288a1f4351930a9ac57305eb1f069db8"
server_key_version: "encryptionkey@0"
I20260925 04:47:38.344990 16665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:47:38.363188 16665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:47:38.366868 16665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:47:38.367082 16665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:47:38.367666 16665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:47:38.368456 16665 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:47:38.368527 16665 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:47:38.368598 16665 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:47:38.368649 16665 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:47:38.377794 16665 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:43709
I20260925 04:47:38.377841 16793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:43709 every 8 connection(s)
I20260925 04:47:38.379053 16665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
I20260925 04:47:38.381367 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 16665
Sep 25 04:47:38 dist-test-slave-kghz krb5kdc[15756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311658, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:47:38.391086 16798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:38.382040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35130 (local address 127.0.98.126:33181)
0925 04:47:38.382296 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0925 04:47:38.382301 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:47:38.382389 (+ 88us) server_negotiation.cc:407] Connection header received
0925 04:47:38.382507 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:47:38.382510 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:47:38.382569 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:47:38.382692 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:47:38.383722 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:47:38.384526 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:47:38.385437 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:47:38.385786 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:47:38.385857 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:47:38.387549 (+ 1692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:47:38.387553 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:47:38.387555 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:47:38.387590 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:47:38.389058 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:47:38.389605 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:47:38.389609 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:47:38.389610 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:47:38.389774 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:47:38.390093 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:47:38.390096 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:47:38.390097 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:47:38.390341 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:47:38.390548 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0925 04:47:38.390716 (+ 168us) server_negotiation.cc:299] Negotiation successful
0925 04:47:38.390852 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":61,"threads_started":1}
I20260925 04:47:38.393930 16526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:38.380895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:34693 (local address 127.0.98.126:33181)
0925 04:47:38.381027 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 04:47:38.381033 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:47:38.381948 (+ 915us) server_negotiation.cc:407] Connection header received
0925 04:47:38.383507 (+ 1559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:47:38.383510 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:47:38.383558 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:47:38.383622 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:47:38.385012 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:47:38.385589 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:47:38.387246 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:47:38.387450 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:47:38.387574 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:47:38.390390 (+ 2816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:47:38.390396 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:47:38.390401 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:47:38.390427 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:47:38.391968 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:47:38.392460 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:47:38.392463 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:47:38.392464 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:47:38.392540 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:47:38.392884 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:47:38.392887 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:47:38.392889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:47:38.393049 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:47:38.393197 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 04:47:38.393682 (+ 485us) server_negotiation.cc:299] Negotiation successful
0925 04:47:38.393787 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 04:47:38.394856 16796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:38.381235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:33181 (local address 127.0.98.67:34693)
0925 04:47:38.381779 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0925 04:47:38.381830 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 04:47:38.383299 (+ 1469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:47:38.383818 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:47:38.383826 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:47:38.384295 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:47:38.384878 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:47:38.384893 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:47:38.385714 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:47:38.385717 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:47:38.387119 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:47:38.387134 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:47:38.387624 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:47:38.387628 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:47:38.387770 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:47:38.388412 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:47:38.388435 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:47:38.390202 (+ 1767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:47:38.392081 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:47:38.392087 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:47:38.392102 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:47:38.392351 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:47:38.392638 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:47:38.392642 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:47:38.392644 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:47:38.392783 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:47:38.393155 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:47:38.393161 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:47:38.393437 (+ 276us) client_negotiation.cc:769] Sending connection context
0925 04:47:38.393646 (+ 209us) client_negotiation.cc:240] Negotiation successful
0925 04:47:38.393973 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":138,"threads_started":1}
I20260925 04:47:38.396173 16794 heartbeater.cc:347] Connected to a master server at 127.0.98.126:33181
I20260925 04:47:38.396432 16794 heartbeater.cc:464] Registering TS with master...
I20260925 04:47:38.396951 16794 heartbeater.cc:511] Master 127.0.98.126:33181 requested a full tablet report, sending...
I20260925 04:47:38.397933 16336 ts_manager.cc:194] Registered new tserver with Master: 0581358747614238835ea146d3d682ea (127.0.98.67:43709)
I20260925 04:47:38.398564 16336 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.67@KRBTEST.COM'} at 127.0.98.67:34693
I20260925 04:47:38.404932 393 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 04:47:38.409209 16526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:38.405636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35132 (local address 127.0.98.126:33181)
0925 04:47:38.405787 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 04:47:38.405792 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:47:38.405986 (+ 194us) server_negotiation.cc:407] Connection header received
0925 04:47:38.406195 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:47:38.406200 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:47:38.406402 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:47:38.406526 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:47:38.406878 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:47:38.407379 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:47:38.408227 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:47:38.408435 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:47:38.408518 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:47:38.408783 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:47:38.408911 (+ 128us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0925 04:47:38.409109 (+ 198us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35132: token expired
Metrics: {"server-negotiator.queue_time_us":52}
W20260925 04:47:38.409294 16526 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35132: token expired
W20260925 04:47:38.409650 16264 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.126:33181)
I20260925 04:47:38.417315 16526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:38.410065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35142 (local address 127.0.98.126:33181)
0925 04:47:38.410172 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0925 04:47:38.410184 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 04:47:38.410199 (+ 15us) server_negotiation.cc:407] Connection header received
0925 04:47:38.410239 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:47:38.410267 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:47:38.410311 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:47:38.410395 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:47:38.411016 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:47:38.411493 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:47:38.412276 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:47:38.412487 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:47:38.412594 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:47:38.414056 (+ 1462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:47:38.414061 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:47:38.414069 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:47:38.414090 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:47:38.415702 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:47:38.416121 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:47:38.416126 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:47:38.416128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:47:38.416209 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:47:38.416487 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:47:38.416491 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:47:38.416493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:47:38.416661 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:47:38.416805 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 04:47:38.417032 (+ 227us) server_negotiation.cc:299] Negotiation successful
0925 04:47:38.417141 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 04:47:38.419857 16336 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35142:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:47:38.423120 16336 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:47:38.430919 16457 tablet_service.cc:1467] Processing CreateTablet for tablet c1e78a8a6a28410284d1bdaddaf7987f (DEFAULT_TABLE table=test-table [id=bbf77cf82ccb44d3aa316bbb734f9b36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:47:38.431604 16457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1e78a8a6a28410284d1bdaddaf7987f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:47:38.434373 16416 tablet_bootstrap.cc:492] T c1e78a8a6a28410284d1bdaddaf7987f P 7a05a5d169f8498a823f5f8b7ac15f92: Bootstrap starting.
I20260925 04:47:38.435261 16416 tablet_bootstrap.cc:654] T c1e78a8a6a28410284d1bdaddaf7987f P 7a05a5d169f8498a823f5f8b7ac15f92: Neither blocks nor log segments found. Creating new log.
I20260925 04:47:38.435753 16531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:38.429927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:43903 (local address 127.0.0.1:48892)
0925 04:47:38.430024 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0925 04:47:38.430042 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 04:47:38.430154 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:47:38.431167 (+ 1013us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:47:38.431170 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:47:38.431190 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:47:38.431320 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:47:38.431329 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:47:38.433021 (+ 1692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:47:38.433028 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:47:38.434269 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:47:38.434279 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:47:38.435291 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:47:38.435294 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:47:38.435388 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:47:38.435431 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 04:47:38.435517 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 04:47:38.435617 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260925 04:47:38.435915 16803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:38.430116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48892 (local address 127.0.98.66:43903)
0925 04:47:38.430601 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0925 04:47:38.430614 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 04:47:38.430638 (+ 24us) server_negotiation.cc:407] Connection header received
0925 04:47:38.430764 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:47:38.430769 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:47:38.430991 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:47:38.431484 (+ 493us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:47:38.431655 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:47:38.432853 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:47:38.434556 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:47:38.435160 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:47:38.435293 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:47:38.435443 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 04:47:38.435667 (+ 224us) server_negotiation.cc:299] Negotiation successful
0925 04:47:38.435767 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":155,"threads_started":1}
I20260925 04:47:38.436023 16804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:38.430496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:43709 (local address 127.0.0.1:37490)
0925 04:47:38.431183 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0925 04:47:38.431197 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:47:38.431301 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:47:38.431796 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:47:38.431799 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:47:38.431818 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:47:38.431908 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:47:38.431914 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:47:38.433354 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:47:38.433357 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:47:38.434147 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:47:38.434155 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:47:38.435554 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:47:38.435559 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:47:38.435680 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:47:38.435715 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 04:47:38.435793 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 04:47:38.435868 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"mutex_wait_us":16,"thread_start_us":91,"threads_started":1}
I20260925 04:47:38.436565 16805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:38.430710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37490 (local address 127.0.98.67:43709)
0925 04:47:38.431098 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0925 04:47:38.431110 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 04:47:38.431304 (+ 194us) server_negotiation.cc:407] Connection header received
0925 04:47:38.431420 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:47:38.431425 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:47:38.431622 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:47:38.431769 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:47:38.432007 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:47:38.433230 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:47:38.434340 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:47:38.435211 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:47:38.435324 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:47:38.435477 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 04:47:38.436252 (+ 775us) server_negotiation.cc:299] Negotiation successful
0925 04:47:38.436375 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":147,"threads_started":1}
I20260925 04:47:38.436851 16416 tablet_bootstrap.cc:492] T c1e78a8a6a28410284d1bdaddaf7987f P 7a05a5d169f8498a823f5f8b7ac15f92: No bootstrap required, opened a new log
I20260925 04:47:38.436950 16416 ts_tablet_manager.cc:1397] T c1e78a8a6a28410284d1bdaddaf7987f P 7a05a5d169f8498a823f5f8b7ac15f92: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 04:47:38.437441 16416 raft_consensus.cc:348] T c1e78a8a6a28410284d1bdaddaf7987f P 7a05a5d169f8498a823f5f8b7ac15f92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c74e7ff9fd45472cbcc9cb1f76994e3e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43903 } } peers { permanent_uuid: "0581358747614238835ea146d3d682ea" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43709 } } peers { permanent_uuid: "7a05a5d169f8498a823f5f8b7ac15f92" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41669 } }
I20260925 04:47:38.437598 16416 raft_consensus.cc:374] T c1e78a8a6a28410284d1bdaddaf7987f P 7a05a5d169f8498a823f5f8b7ac15f92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:47:38.437642 16416 raft_consensus.cc:729] T c1e78a8a6a28410284d1bdaddaf7987f P 7a05a5d169f8498a823f5f8b7ac15f92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a05a5d169f8498a823f5f8b7ac15f92, State: Initialized, Role: FOLLOWER
I20260925 04:47:38.437785 16416 consensus_queue.cc:260] T c1e78a8a6a28410284d1bdaddaf7987f P 7a05a5d169f8498a823f5f8b7ac15f92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c74e7ff9fd45472cbcc9cb1f76994e3e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43903 } } peers { permanent_uuid: "0581358747614238835ea146d3d682ea" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43709 } } peers { permanent_uuid: "7a05a5d169f8498a823f5f8b7ac15f92" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41669 } }
I20260925 04:47:38.438067 16416 ts_tablet_manager.cc:1428] T c1e78a8a6a28410284d1bdaddaf7987f P 7a05a5d169f8498a823f5f8b7ac15f92: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 04:47:38.439481 16596 tablet_service.cc:1467] Processing CreateTablet for tablet c1e78a8a6a28410284d1bdaddaf7987f (DEFAULT_TABLE table=test-table [id=bbf77cf82ccb44d3aa316bbb734f9b36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:47:38.439482 16728 tablet_service.cc:1467] Processing CreateTablet for tablet c1e78a8a6a28410284d1bdaddaf7987f (DEFAULT_TABLE table=test-table [id=bbf77cf82ccb44d3aa316bbb734f9b36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:47:38.440640 16728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1e78a8a6a28410284d1bdaddaf7987f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:47:38.440637 16596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1e78a8a6a28410284d1bdaddaf7987f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:47:38.449092 16807 tablet_bootstrap.cc:492] T c1e78a8a6a28410284d1bdaddaf7987f P 0581358747614238835ea146d3d682ea: Bootstrap starting.
I20260925 04:47:38.449630 16808 tablet_bootstrap.cc:492] T c1e78a8a6a28410284d1bdaddaf7987f P c74e7ff9fd45472cbcc9cb1f76994e3e: Bootstrap starting.
I20260925 04:47:38.452713 16807 tablet_bootstrap.cc:654] T c1e78a8a6a28410284d1bdaddaf7987f P 0581358747614238835ea146d3d682ea: Neither blocks nor log segments found. Creating new log.
I20260925 04:47:38.452713 16808 tablet_bootstrap.cc:654] T c1e78a8a6a28410284d1bdaddaf7987f P c74e7ff9fd45472cbcc9cb1f76994e3e: Neither blocks nor log segments found. Creating new log.
I20260925 04:47:38.453672 16807 log.cc:824] T c1e78a8a6a28410284d1bdaddaf7987f P 0581358747614238835ea146d3d682ea: Log is configured to *not* fsync() on all Append() calls
I20260925 04:47:38.453672 16808 log.cc:824] T c1e78a8a6a28410284d1bdaddaf7987f P c74e7ff9fd45472cbcc9cb1f76994e3e: Log is configured to *not* fsync() on all Append() calls
I20260925 04:47:38.455624 16808 tablet_bootstrap.cc:492] T c1e78a8a6a28410284d1bdaddaf7987f P c74e7ff9fd45472cbcc9cb1f76994e3e: No bootstrap required, opened a new log
I20260925 04:47:38.455641 16807 tablet_bootstrap.cc:492] T c1e78a8a6a28410284d1bdaddaf7987f P 0581358747614238835ea146d3d682ea: No bootstrap required, opened a new log
I20260925 04:47:38.455864 16808 ts_tablet_manager.cc:1397] T c1e78a8a6a28410284d1bdaddaf7987f P c74e7ff9fd45472cbcc9cb1f76994e3e: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 04:47:38.455936 16807 ts_tablet_manager.cc:1397] T c1e78a8a6a28410284d1bdaddaf7987f P 0581358747614238835ea146d3d682ea: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260925 04:47:38.459115 16808 raft_consensus.cc:348] T c1e78a8a6a28410284d1bdaddaf7987f P c74e7ff9fd45472cbcc9cb1f76994e3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c74e7ff9fd45472cbcc9cb1f76994e3e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43903 } } peers { permanent_uuid: "0581358747614238835ea146d3d682ea" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43709 } } peers { permanent_uuid: "7a05a5d169f8498a823f5f8b7ac15f92" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41669 } }
I20260925 04:47:38.459354 16808 raft_consensus.cc:374] T c1e78a8a6a28410284d1bdaddaf7987f P c74e7ff9fd45472cbcc9cb1f76994e3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:47:38.459208 16807 raft_consensus.cc:348] T c1e78a8a6a28410284d1bdaddaf7987f P 0581358747614238835ea146d3d682ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c74e7ff9fd45472cbcc9cb1f76994e3e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43903 } } peers { permanent_uuid: "0581358747614238835ea146d3d682ea" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43709 } } peers { permanent_uuid: "7a05a5d169f8498a823f5f8b7ac15f92" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41669 } }
I20260925 04:47:38.459388 16808 raft_consensus.cc:729] T c1e78a8a6a28410284d1bdaddaf7987f P c74e7ff9fd45472cbcc9cb1f76994e3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c74e7ff9fd45472cbcc9cb1f76994e3e, State: Initialized, Role: FOLLOWER
I20260925 04:47:38.459472 16807 raft_consensus.cc:374] T c1e78a8a6a28410284d1bdaddaf7987f P 0581358747614238835ea146d3d682ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:47:38.459530 16807 raft_consensus.cc:729] T c1e78a8a6a28410284d1bdaddaf7987f P 0581358747614238835ea146d3d682ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0581358747614238835ea146d3d682ea, State: Initialized, Role: FOLLOWER
I20260925 04:47:38.459863 16808 consensus_queue.cc:260] T c1e78a8a6a28410284d1bdaddaf7987f P c74e7ff9fd45472cbcc9cb1f76994e3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c74e7ff9fd45472cbcc9cb1f76994e3e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43903 } } peers { permanent_uuid: "0581358747614238835ea146d3d682ea" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43709 } } peers { permanent_uuid: "7a05a5d169f8498a823f5f8b7ac15f92" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41669 } }
I20260925 04:47:38.459928 16807 consensus_queue.cc:260] T c1e78a8a6a28410284d1bdaddaf7987f P 0581358747614238835ea146d3d682ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c74e7ff9fd45472cbcc9cb1f76994e3e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43903 } } peers { permanent_uuid: "0581358747614238835ea146d3d682ea" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43709 } } peers { permanent_uuid: "7a05a5d169f8498a823f5f8b7ac15f92" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41669 } }
I20260925 04:47:38.460549 16662 heartbeater.cc:503] Master 127.0.98.126:33181 was elected leader, sending a full tablet report...
I20260925 04:47:38.460599 16794 heartbeater.cc:503] Master 127.0.98.126:33181 was elected leader, sending a full tablet report...
I20260925 04:47:38.460932 16807 ts_tablet_manager.cc:1428] T c1e78a8a6a28410284d1bdaddaf7987f P 0581358747614238835ea146d3d682ea: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260925 04:47:38.460940 16808 ts_tablet_manager.cc:1428] T c1e78a8a6a28410284d1bdaddaf7987f P c74e7ff9fd45472cbcc9cb1f76994e3e: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260925 04:47:38.473186 16812 raft_consensus.cc:482] T c1e78a8a6a28410284d1bdaddaf7987f P 0581358747614238835ea146d3d682ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:47:38.473362 16812 raft_consensus.cc:504] T c1e78a8a6a28410284d1bdaddaf7987f P 0581358747614238835ea146d3d682ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c74e7ff9fd45472cbcc9cb1f76994e3e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43903 } } peers { permanent_uuid: "0581358747614238835ea146d3d682ea" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43709 } } peers { permanent_uuid: "7a05a5d169f8498a823f5f8b7ac15f92" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41669 } }
I20260925 04:47:38.474490 16812 leader_election.cc:288] T c1e78a8a6a28410284d1bdaddaf7987f P 0581358747614238835ea146d3d682ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c74e7ff9fd45472cbcc9cb1f76994e3e (127.0.98.66:43903), 7a05a5d169f8498a823f5f8b7ac15f92 (127.0.98.65:41669)
I20260925 04:47:38.478088 16532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:38.474865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:52463 (local address 127.0.98.65:41669)
0925 04:47:38.474980 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0925 04:47:38.474984 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:47:38.475000 (+ 16us) server_negotiation.cc:407] Connection header received
0925 04:47:38.475092 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:47:38.475095 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:47:38.475157 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:47:38.475267 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:47:38.475536 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:47:38.476081 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:47:38.477099 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:47:38.477547 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:47:38.477688 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:47:38.477861 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0925 04:47:38.477901 (+ 40us) server_negotiation.cc:299] Negotiation successful
0925 04:47:38.477966 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 04:47:38.478112 16796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:38.474721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:41669 (local address 127.0.98.67:52463)
0925 04:47:38.474893 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 04:47:38.474911 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 04:47:38.474996 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:47:38.475275 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:47:38.475279 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:47:38.475295 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:47:38.475418 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:47:38.475424 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:47:38.476198 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:47:38.476204 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:47:38.476969 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:47:38.476978 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:47:38.477688 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:47:38.477692 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:47:38.477797 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:47:38.477818 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 04:47:38.477887 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 04:47:38.477961 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95}
I20260925 04:47:38.478229 16803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:38.474805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:48085 (local address 127.0.98.66:43903)
0925 04:47:38.474946 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 04:47:38.474951 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:47:38.475196 (+ 245us) server_negotiation.cc:407] Connection header received
0925 04:47:38.475349 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:47:38.475354 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:47:38.475401 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:47:38.475485 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:47:38.475749 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:47:38.476226 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:47:38.477239 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:47:38.477645 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:47:38.477767 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:47:38.477827 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 04:47:38.478014 (+ 187us) server_negotiation.cc:299] Negotiation successful
0925 04:47:38.478084 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 04:47:38.478565 16813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:38.474770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:43903 (local address 127.0.98.67:48085)
0925 04:47:38.475091 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0925 04:47:38.475102 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 04:47:38.475236 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:47:38.475494 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:47:38.475496 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:47:38.475511 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:47:38.475637 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:47:38.475642 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:47:38.476338 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:47:38.476343 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:47:38.477110 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:47:38.477118 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:47:38.477771 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:47:38.477775 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:47:38.477881 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:47:38.477903 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 04:47:38.478000 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 04:47:38.478091 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":124,"threads_started":1}
I20260925 04:47:38.478920 16478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1e78a8a6a28410284d1bdaddaf7987f" candidate_uuid: "0581358747614238835ea146d3d682ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a05a5d169f8498a823f5f8b7ac15f92" is_pre_election: true
I20260925 04:47:38.479212 16616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1e78a8a6a28410284d1bdaddaf7987f" candidate_uuid: "0581358747614238835ea146d3d682ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c74e7ff9fd45472cbcc9cb1f76994e3e" is_pre_election: true
I20260925 04:47:38.479251 16478 raft_consensus.cc:2454] T c1e78a8a6a28410284d1bdaddaf7987f P 7a05a5d169f8498a823f5f8b7ac15f92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0581358747614238835ea146d3d682ea in term 0.
I20260925 04:47:38.479513 16616 raft_consensus.cc:2454] T c1e78a8a6a28410284d1bdaddaf7987f P c74e7ff9fd45472cbcc9cb1f76994e3e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0581358747614238835ea146d3d682ea in term 0.
I20260925 04:47:38.479763 16682 leader_election.cc:302] T c1e78a8a6a28410284d1bdaddaf7987f P 0581358747614238835ea146d3d682ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0581358747614238835ea146d3d682ea, 7a05a5d169f8498a823f5f8b7ac15f92; no voters:
I20260925 04:47:38.480041 16812 raft_consensus.cc:2781] T c1e78a8a6a28410284d1bdaddaf7987f P 0581358747614238835ea146d3d682ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:47:38.480134 16812 raft_consensus.cc:482] T c1e78a8a6a28410284d1bdaddaf7987f P 0581358747614238835ea146d3d682ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:47:38.480178 16812 raft_consensus.cc:3037] T c1e78a8a6a28410284d1bdaddaf7987f P 0581358747614238835ea146d3d682ea [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:47:38.481212 16812 raft_consensus.cc:504] T c1e78a8a6a28410284d1bdaddaf7987f P 0581358747614238835ea146d3d682ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c74e7ff9fd45472cbcc9cb1f76994e3e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43903 } } peers { permanent_uuid: "0581358747614238835ea146d3d682ea" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43709 } } peers { permanent_uuid: "7a05a5d169f8498a823f5f8b7ac15f92" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41669 } }
I20260925 04:47:38.481604 16812 leader_election.cc:288] T c1e78a8a6a28410284d1bdaddaf7987f P 0581358747614238835ea146d3d682ea [CANDIDATE]: Term 1 election: Requested vote from peers c74e7ff9fd45472cbcc9cb1f76994e3e (127.0.98.66:43903), 7a05a5d169f8498a823f5f8b7ac15f92 (127.0.98.65:41669)
I20260925 04:47:38.481961 16616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1e78a8a6a28410284d1bdaddaf7987f" candidate_uuid: "0581358747614238835ea146d3d682ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c74e7ff9fd45472cbcc9cb1f76994e3e"
I20260925 04:47:38.481972 16478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1e78a8a6a28410284d1bdaddaf7987f" candidate_uuid: "0581358747614238835ea146d3d682ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a05a5d169f8498a823f5f8b7ac15f92"
I20260925 04:47:38.482120 16478 raft_consensus.cc:3037] T c1e78a8a6a28410284d1bdaddaf7987f P 7a05a5d169f8498a823f5f8b7ac15f92 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:47:38.482122 16616 raft_consensus.cc:3037] T c1e78a8a6a28410284d1bdaddaf7987f P c74e7ff9fd45472cbcc9cb1f76994e3e [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:47:38.483237 16616 raft_consensus.cc:2454] T c1e78a8a6a28410284d1bdaddaf7987f P c74e7ff9fd45472cbcc9cb1f76994e3e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0581358747614238835ea146d3d682ea in term 1.
I20260925 04:47:38.483261 16478 raft_consensus.cc:2454] T c1e78a8a6a28410284d1bdaddaf7987f P 7a05a5d169f8498a823f5f8b7ac15f92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0581358747614238835ea146d3d682ea in term 1.
I20260925 04:47:38.483624 16681 leader_election.cc:302] T c1e78a8a6a28410284d1bdaddaf7987f P 0581358747614238835ea146d3d682ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0581358747614238835ea146d3d682ea, c74e7ff9fd45472cbcc9cb1f76994e3e; no voters:
I20260925 04:47:38.483827 16812 raft_consensus.cc:2781] T c1e78a8a6a28410284d1bdaddaf7987f P 0581358747614238835ea146d3d682ea [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:47:38.484081 16812 raft_consensus.cc:686] T c1e78a8a6a28410284d1bdaddaf7987f P 0581358747614238835ea146d3d682ea [term 1 LEADER]: Becoming Leader. State: Replica: 0581358747614238835ea146d3d682ea, State: Running, Role: LEADER
I20260925 04:47:38.484473 16812 consensus_queue.cc:237] T c1e78a8a6a28410284d1bdaddaf7987f P 0581358747614238835ea146d3d682ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c74e7ff9fd45472cbcc9cb1f76994e3e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43903 } } peers { permanent_uuid: "0581358747614238835ea146d3d682ea" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43709 } } peers { permanent_uuid: "7a05a5d169f8498a823f5f8b7ac15f92" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41669 } }
I20260925 04:47:38.487627 16335 catalog_manager.cc:5495] T c1e78a8a6a28410284d1bdaddaf7987f P 0581358747614238835ea146d3d682ea reported cstate change: term changed from 0 to 1, leader changed from <none> to 0581358747614238835ea146d3d682ea (127.0.98.67). New cstate: current_term: 1 leader_uuid: "0581358747614238835ea146d3d682ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c74e7ff9fd45472cbcc9cb1f76994e3e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0581358747614238835ea146d3d682ea" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a05a5d169f8498a823f5f8b7ac15f92" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41669 } health_report { overall_health: UNKNOWN } } }
W20260925 04:47:38.502605 16525 tablet.cc:2369] T c1e78a8a6a28410284d1bdaddaf7987f P 7a05a5d169f8498a823f5f8b7ac15f92: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:47:38.502853 16805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:38.499258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37506 (local address 127.0.98.67:43709)
0925 04:47:38.499394 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 04:47:38.499401 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:47:38.499417 (+ 16us) server_negotiation.cc:407] Connection header received
0925 04:47:38.499606 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:47:38.499608 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:47:38.499662 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:47:38.499773 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:47:38.500021 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:47:38.500529 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:47:38.501352 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:47:38.501574 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:47:38.501684 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:47:38.501979 (+ 295us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:47:38.502434 (+ 455us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:47:38.502563 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 04:47:38.502622 (+ 59us) server_negotiation.cc:299] Negotiation successful
0925 04:47:38.502679 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 04:47:38.508345 16478 raft_consensus.cc:1261] T c1e78a8a6a28410284d1bdaddaf7987f P 7a05a5d169f8498a823f5f8b7ac15f92 [term 1 FOLLOWER]: Refusing update from remote peer 0581358747614238835ea146d3d682ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:47:38.508349 16616 raft_consensus.cc:1261] T c1e78a8a6a28410284d1bdaddaf7987f P c74e7ff9fd45472cbcc9cb1f76994e3e [term 1 FOLLOWER]: Refusing update from remote peer 0581358747614238835ea146d3d682ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:47:38.509023 16812 consensus_queue.cc:1035] T c1e78a8a6a28410284d1bdaddaf7987f P 0581358747614238835ea146d3d682ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a05a5d169f8498a823f5f8b7ac15f92" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:47:38.509241 16814 consensus_queue.cc:1035] T c1e78a8a6a28410284d1bdaddaf7987f P 0581358747614238835ea146d3d682ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "c74e7ff9fd45472cbcc9cb1f76994e3e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:47:38.514447 16817 mvcc.cc:205] Tried to move back new op lower bound from 7333116553243435008 to 7333116553155465216. Current Snapshot: MvccSnapshot[applied={T|T < 7333116553243435008}]
I20260925 04:47:38.515703 16818 mvcc.cc:205] Tried to move back new op lower bound from 7333116553243435008 to 7333116553155465216. Current Snapshot: MvccSnapshot[applied={T|T < 7333116553243435008}]
I20260925 04:47:38.523128 16336 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35142:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:47:38.523312 16336 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35142:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:47:38.525552 16336 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a175a4ff2ea44c5ca7a8f69fec22566a: Sending DeleteTablet for 3 replicas of tablet c1e78a8a6a28410284d1bdaddaf7987f
I20260925 04:47:38.526304 16457 tablet_service.cc:1514] Processing DeleteTablet for tablet c1e78a8a6a28410284d1bdaddaf7987f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:47:38 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:34134
I20260925 04:47:38.526369 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 16374
I20260925 04:47:38.526384 16728 tablet_service.cc:1514] Processing DeleteTablet for tablet c1e78a8a6a28410284d1bdaddaf7987f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:47:38 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:37490
I20260925 04:47:38.526494 16533 tablet_replica.cc:330] T c1e78a8a6a28410284d1bdaddaf7987f P 7a05a5d169f8498a823f5f8b7ac15f92: stopping tablet replica
I20260925 04:47:38.526600 16533 raft_consensus.cc:2229] T c1e78a8a6a28410284d1bdaddaf7987f P 7a05a5d169f8498a823f5f8b7ac15f92 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:47:38.527215 16596 tablet_service.cc:1514] Processing DeleteTablet for tablet c1e78a8a6a28410284d1bdaddaf7987f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:47:38 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:48892
I20260925 04:47:38.527663 16828 tablet_replica.cc:330] T c1e78a8a6a28410284d1bdaddaf7987f P c74e7ff9fd45472cbcc9cb1f76994e3e: stopping tablet replica
I20260925 04:47:38.527709 16827 tablet_replica.cc:330] T c1e78a8a6a28410284d1bdaddaf7987f P 0581358747614238835ea146d3d682ea: stopping tablet replica
I20260925 04:47:38.527992 16827 raft_consensus.cc:2229] T c1e78a8a6a28410284d1bdaddaf7987f P 0581358747614238835ea146d3d682ea [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:47:38.527993 16828 raft_consensus.cc:2229] T c1e78a8a6a28410284d1bdaddaf7987f P c74e7ff9fd45472cbcc9cb1f76994e3e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:47:38.528280 16828 raft_consensus.cc:2258] T c1e78a8a6a28410284d1bdaddaf7987f P c74e7ff9fd45472cbcc9cb1f76994e3e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:47:38.528383 16827 raft_consensus.cc:2258] T c1e78a8a6a28410284d1bdaddaf7987f P 0581358747614238835ea146d3d682ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:47:38.529724 16827 ts_tablet_manager.cc:1905] T c1e78a8a6a28410284d1bdaddaf7987f P 0581358747614238835ea146d3d682ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:47:38.529930 16828 ts_tablet_manager.cc:1905] T c1e78a8a6a28410284d1bdaddaf7987f P c74e7ff9fd45472cbcc9cb1f76994e3e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:47:38.532603 16827 ts_tablet_manager.cc:1918] T c1e78a8a6a28410284d1bdaddaf7987f P 0581358747614238835ea146d3d682ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:47:38.532603 16828 ts_tablet_manager.cc:1918] T c1e78a8a6a28410284d1bdaddaf7987f P c74e7ff9fd45472cbcc9cb1f76994e3e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:47:38.532702 16828 log.cc:1196] T c1e78a8a6a28410284d1bdaddaf7987f P c74e7ff9fd45472cbcc9cb1f76994e3e: Deleting WAL directory at /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-1/wal/wals/c1e78a8a6a28410284d1bdaddaf7987f
I20260925 04:47:38.532812 16827 log.cc:1196] T c1e78a8a6a28410284d1bdaddaf7987f P 0581358747614238835ea146d3d682ea: Deleting WAL directory at /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-2/wal/wals/c1e78a8a6a28410284d1bdaddaf7987f
I20260925 04:47:38.533082 16828 ts_tablet_manager.cc:1939] T c1e78a8a6a28410284d1bdaddaf7987f P c74e7ff9fd45472cbcc9cb1f76994e3e: Deleting consensus metadata
I20260925 04:47:38.533243 16827 ts_tablet_manager.cc:1939] T c1e78a8a6a28410284d1bdaddaf7987f P 0581358747614238835ea146d3d682ea: Deleting consensus metadata
I20260925 04:47:38.534406 16320 catalog_manager.cc:4841] TS c74e7ff9fd45472cbcc9cb1f76994e3e (127.0.98.66:43903): tablet c1e78a8a6a28410284d1bdaddaf7987f (table test-table [id=bbf77cf82ccb44d3aa316bbb734f9b36]) successfully deleted
I20260925 04:47:38.534412 16323 catalog_manager.cc:4841] TS 0581358747614238835ea146d3d682ea (127.0.98.67:43709): tablet c1e78a8a6a28410284d1bdaddaf7987f (table test-table [id=bbf77cf82ccb44d3aa316bbb734f9b36]) successfully deleted
W20260925 04:47:38.534811 16321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.65:41669 (error 108)
I20260925 04:47:38.535040 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 16528
I20260925 04:47:38.537353 16804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:38.537056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:41669 (local address 127.0.0.1:34146)
0925 04:47:38.537228 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 04:47:38.537287 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.65:41669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
W20260925 04:47:38.537549 16321 catalog_manager.cc:4584] TS 7a05a5d169f8498a823f5f8b7ac15f92 (127.0.98.65:41669): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c1e78a8a6a28410284d1bdaddaf7987f: Network error: Client connection negotiation failed: client connection to 127.0.98.65:41669: connect: Connection refused (error 111)
I20260925 04:47:38.542608 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 16665
I20260925 04:47:38.548724 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 16303
I20260925 04:47:58.555548 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:33181
--webserver_interface=127.0.98.126
--webserver_port=38533
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:39863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:33181
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:47:58.673040 16830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:47:58.673309 16830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:47:58.673383 16830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:47:58.677415 16830 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 04:47:58.677471 16830 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 04:47:58.677486 16830 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 04:47:58.677500 16830 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:47:58.677548 16830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:47:58.677584 16830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:47:58.677631 16830 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:47:58.677670 16830 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:47:58.682960 16830 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:39863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:33181
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:33181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=38533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:47:58.684017 16830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:47:58.685101 16830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:47:58.693938 16838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:47:58.693970 16835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:47:58.693967 16836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:47:58.694218 16830 server_base.cc:1034] running on GCE node
I20260925 04:47:58.694857 16830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:47:58.695916 16830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:47:58.697124 16830 hybrid_clock.cc:648] HybridClock initialized: now 1790311678697100 us; error 40 us; skew 500 ppm
Sep 25 04:47:58 dist-test-slave-kghz krb5kdc[15756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311678, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:47:58.700001 16830 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260925 04:47:58.700904 16830 webserver.cc:460] Webserver started at http://127.0.98.126:38533/ using document root <none> and password file <none>
I20260925 04:47:58.701471 16830 fs_manager.cc:359] Metadata directory not provided
I20260925 04:47:58.701557 16830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:47:58.705507 16830 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260925 04:47:58.707710 16845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:47:58.708973 16830 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 04:47:58.709136 16830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/master-0/wal
uuid: "a175a4ff2ea44c5ca7a8f69fec22566a"
format_stamp: "Formatted at 2026-09-25 04:47:16 on dist-test-slave-kghz"
server_key: "184464def56b690e2a70b606620f3760"
server_key_iv: "b9c5f2a8beaab36785dae5e8815eeff3"
server_key_version: "encryptionkey@0"
I20260925 04:47:58.709705 16830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:47:58.724690 16830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:47:58.727806 16830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:47:58.728005 16830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:47:58.735035 16830 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:33181
I20260925 04:47:58.735323 16897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:33181 every 8 connection(s)
I20260925 04:47:58.736141 16830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
I20260925 04:47:58.740471 16898 sys_catalog.cc:263] Verifying existing consensus state
I20260925 04:47:58.742210 16898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a175a4ff2ea44c5ca7a8f69fec22566a: Bootstrap starting.
I20260925 04:47:58.742383 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 16830
I20260925 04:47:58.742820 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:41669
--local_ip_for_outbound_sockets=127.0.98.65
--tserver_master_addrs=127.0.98.126:33181
--webserver_port=38693
--webserver_interface=127.0.98.65
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:39863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 04:47:58.760114 16898 log.cc:824] T 00000000000000000000000000000000 P a175a4ff2ea44c5ca7a8f69fec22566a: Log is configured to *not* fsync() on all Append() calls
I20260925 04:47:58.771512 16898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a175a4ff2ea44c5ca7a8f69fec22566a: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 04:47:58.771925 16898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a175a4ff2ea44c5ca7a8f69fec22566a: Bootstrap complete.
I20260925 04:47:58.775946 16898 raft_consensus.cc:348] T 00000000000000000000000000000000 P a175a4ff2ea44c5ca7a8f69fec22566a [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a175a4ff2ea44c5ca7a8f69fec22566a" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33181 } }
I20260925 04:47:58.776319 16898 raft_consensus.cc:729] T 00000000000000000000000000000000 P a175a4ff2ea44c5ca7a8f69fec22566a [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: a175a4ff2ea44c5ca7a8f69fec22566a, State: Initialized, Role: FOLLOWER
I20260925 04:47:58.776796 16898 consensus_queue.cc:260] T 00000000000000000000000000000000 P a175a4ff2ea44c5ca7a8f69fec22566a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a175a4ff2ea44c5ca7a8f69fec22566a" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33181 } }
I20260925 04:47:58.776906 16898 raft_consensus.cc:388] T 00000000000000000000000000000000 P a175a4ff2ea44c5ca7a8f69fec22566a [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:47:58.776991 16898 raft_consensus.cc:482] T 00000000000000000000000000000000 P a175a4ff2ea44c5ca7a8f69fec22566a [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:47:58.777097 16898 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a175a4ff2ea44c5ca7a8f69fec22566a [term 2 FOLLOWER]: Advancing to term 3
I20260925 04:47:58.779057 16898 raft_consensus.cc:504] T 00000000000000000000000000000000 P a175a4ff2ea44c5ca7a8f69fec22566a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a175a4ff2ea44c5ca7a8f69fec22566a" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33181 } }
I20260925 04:47:58.779392 16898 leader_election.cc:302] T 00000000000000000000000000000000 P a175a4ff2ea44c5ca7a8f69fec22566a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a175a4ff2ea44c5ca7a8f69fec22566a; no voters:
I20260925 04:47:58.779723 16898 leader_election.cc:288] T 00000000000000000000000000000000 P a175a4ff2ea44c5ca7a8f69fec22566a [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 04:47:58.779829 16902 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a175a4ff2ea44c5ca7a8f69fec22566a [term 3 FOLLOWER]: Leader election won for term 3
I20260925 04:47:58.780021 16902 raft_consensus.cc:686] T 00000000000000000000000000000000 P a175a4ff2ea44c5ca7a8f69fec22566a [term 3 LEADER]: Becoming Leader. State: Replica: a175a4ff2ea44c5ca7a8f69fec22566a, State: Running, Role: LEADER
I20260925 04:47:58.780380 16902 consensus_queue.cc:237] T 00000000000000000000000000000000 P a175a4ff2ea44c5ca7a8f69fec22566a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a175a4ff2ea44c5ca7a8f69fec22566a" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33181 } }
I20260925 04:47:58.781013 16898 sys_catalog.cc:564] T 00000000000000000000000000000000 P a175a4ff2ea44c5ca7a8f69fec22566a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:47:58.781733 16903 sys_catalog.cc:455] T 00000000000000000000000000000000 P a175a4ff2ea44c5ca7a8f69fec22566a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "a175a4ff2ea44c5ca7a8f69fec22566a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a175a4ff2ea44c5ca7a8f69fec22566a" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33181 } } }
I20260925 04:47:58.782158 16903 sys_catalog.cc:458] T 00000000000000000000000000000000 P a175a4ff2ea44c5ca7a8f69fec22566a [sys.catalog]: This master's current role is: LEADER
I20260925 04:47:58.782063 16904 sys_catalog.cc:455] T 00000000000000000000000000000000 P a175a4ff2ea44c5ca7a8f69fec22566a [sys.catalog]: SysCatalogTable state changed. Reason: New leader a175a4ff2ea44c5ca7a8f69fec22566a. Latest consensus state: current_term: 3 leader_uuid: "a175a4ff2ea44c5ca7a8f69fec22566a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a175a4ff2ea44c5ca7a8f69fec22566a" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33181 } } }
I20260925 04:47:58.782498 16904 sys_catalog.cc:458] T 00000000000000000000000000000000 P a175a4ff2ea44c5ca7a8f69fec22566a [sys.catalog]: This master's current role is: LEADER
I20260925 04:47:58.782799 16912 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:47:58.788174 16912 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:47:58.789001 16912 catalog_manager.cc:1259] Loaded cluster ID: c272a46e210846a7a5d01e8e195412a3
I20260925 04:47:58.789064 16912 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:47:58.791827 16912 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:47:58.792536 16912 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a175a4ff2ea44c5ca7a8f69fec22566a: Loaded TSK: 0
I20260925 04:47:58.792616 16912 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a175a4ff2ea44c5ca7a8f69fec22566a: Loaded TSK: 1
F20260925 04:47:58.793316 16912 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f4c19b139ad google::LogMessage::Fail() at ??:0
@ 0x7f4c19b1688a google::LogMessage::SendToLog() at ??:0
@ 0x7f4c19b134de google::LogMessage::Flush() at ??:0
@ 0x7f4c19b14a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f4c1c7d718d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f4c1c7cff0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f4c244787bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f4c2445299c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f4c24453178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f4c24494df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f4c244b382b std::function<>::operator()() at ??:0
@ 0x7f4c24452b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f4c24453f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f4c24450579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f4c24494640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f4c1bd653d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f4c1bd65c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f4c1bd673f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f4c1bd56512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f4c1d39a6db start_thread at ??:0
@ 0x7f4c1861161f clone at ??:0
W20260925 04:47:58.895618 16900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:47:58.895889 16900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:47:58.895933 16900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:47:58.900110 16900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:47:58.900177 16900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:47:58.900266 16900 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260925 04:47:58.905648 16900 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:39863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:41669
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=38693
--tserver_master_addrs=127.0.98.126:33181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16900
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:47:58.906762 16900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:47:58.907779 16900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:47:58.917464 16946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:47:58.917450 16948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:47:58.917457 16945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:47:58.917780 16900 server_base.cc:1034] running on GCE node
I20260925 04:47:58.918303 16900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:47:58.919016 16900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:47:58.920199 16900 hybrid_clock.cc:648] HybridClock initialized: now 1790311678920163 us; error 59 us; skew 500 ppm
Sep 25 04:47:58 dist-test-slave-kghz krb5kdc[15756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311678, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:47:58.926069 16900 init.cc:377] Logged in from keytab as kudu/127.0.98.65@KRBTEST.COM (short username kudu)
I20260925 04:47:58.927059 16900 webserver.cc:460] Webserver started at http://127.0.98.65:38693/ using document root <none> and password file <none>
I20260925 04:47:58.927647 16900 fs_manager.cc:359] Metadata directory not provided
I20260925 04:47:58.927740 16900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:47:58.931900 16900 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.002s
I20260925 04:47:58.934218 16955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:47:58.935552 16900 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 04:47:58.935693 16900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-0/wal
uuid: "7a05a5d169f8498a823f5f8b7ac15f92"
format_stamp: "Formatted at 2026-09-25 04:47:16 on dist-test-slave-kghz"
server_key: "42d5f01b4c6f1a6ccbf7cd191e555c7d"
server_key_iv: "b2d33c06103ea5639fe80b0a86c5cfd6"
server_key_version: "encryptionkey@0"
I20260925 04:47:58.936259 16900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:47:58.962474 16900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:47:58.966396 16900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:47:58.966645 16900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:47:58.967430 16900 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:47:58.968626 16962 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 04:47:58.972759 16900 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 04:47:58.972824 16900 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260925 04:47:58.972889 16900 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 04:47:58.975077 16900 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 04:47:58.975136 16900 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260925 04:47:58.975382 16962 tablet_bootstrap.cc:492] T c1e78a8a6a28410284d1bdaddaf7987f P 7a05a5d169f8498a823f5f8b7ac15f92: Bootstrap starting.
I20260925 04:47:58.986188 16900 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:41669
I20260925 04:47:58.986443 17069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:41669 every 8 connection(s)
I20260925 04:47:58.987610 16900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
I20260925 04:47:58.990536 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 16900
I20260925 04:47:58.991055 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:43903
--local_ip_for_outbound_sockets=127.0.98.66
--tserver_master_addrs=127.0.98.126:33181
--webserver_port=33783
--webserver_interface=127.0.98.66
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:39863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 04:47:58.997357 16962 log.cc:824] T c1e78a8a6a28410284d1bdaddaf7987f P 7a05a5d169f8498a823f5f8b7ac15f92: Log is configured to *not* fsync() on all Append() calls
I20260925 04:47:59.006457 16962 tablet_bootstrap.cc:492] T c1e78a8a6a28410284d1bdaddaf7987f P 7a05a5d169f8498a823f5f8b7ac15f92: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 04:47:59.006925 16962 tablet_bootstrap.cc:492] T c1e78a8a6a28410284d1bdaddaf7987f P 7a05a5d169f8498a823f5f8b7ac15f92: Bootstrap complete.
I20260925 04:47:59.008342 16962 ts_tablet_manager.cc:1397] T c1e78a8a6a28410284d1bdaddaf7987f P 7a05a5d169f8498a823f5f8b7ac15f92: Time spent bootstrapping tablet: real 0.033s user 0.026s sys 0.004s
I20260925 04:47:59.011727 16962 raft_consensus.cc:348] T c1e78a8a6a28410284d1bdaddaf7987f P 7a05a5d169f8498a823f5f8b7ac15f92 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c74e7ff9fd45472cbcc9cb1f76994e3e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43903 } } peers { permanent_uuid: "0581358747614238835ea146d3d682ea" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43709 } } peers { permanent_uuid: "7a05a5d169f8498a823f5f8b7ac15f92" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41669 } }
I20260925 04:47:59.012279 16962 raft_consensus.cc:729] T c1e78a8a6a28410284d1bdaddaf7987f P 7a05a5d169f8498a823f5f8b7ac15f92 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a05a5d169f8498a823f5f8b7ac15f92, State: Initialized, Role: FOLLOWER
I20260925 04:47:59.012830 16962 consensus_queue.cc:260] T c1e78a8a6a28410284d1bdaddaf7987f P 7a05a5d169f8498a823f5f8b7ac15f92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c74e7ff9fd45472cbcc9cb1f76994e3e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43903 } } peers { permanent_uuid: "0581358747614238835ea146d3d682ea" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43709 } } peers { permanent_uuid: "7a05a5d169f8498a823f5f8b7ac15f92" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41669 } }
I20260925 04:47:59.013875 16962 ts_tablet_manager.cc:1428] T c1e78a8a6a28410284d1bdaddaf7987f P 7a05a5d169f8498a823f5f8b7ac15f92: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
W20260925 04:47:59.025796 17072 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:47:58.989950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:33181 (local address 127.0.98.65:40943)
0925 04:47:58.990602 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0925 04:47:58.990680 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0925 04:47:58.991951 (+ 1271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:47:59.025065 (+ 33114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:33181: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":204,"threads_started":1}
W20260925 04:47:59.026335 17070 heartbeater.cc:644] Failed to heartbeat to 127.0.98.126:33181 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.126:33181: Client connection negotiation failed: client connection to 127.0.98.126:33181: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 04:47:59.027066 17072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:59.026831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:33181 (local address 127.0.98.65:32965)
0925 04:47:59.026953 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 04:47:59.027013 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260925 04:47:59.028220 17072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:59.028049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:33181 (local address 127.0.98.65:36157)
0925 04:47:59.028140 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0925 04:47:59.028178 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260925 04:47:59.028447 17070 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 04:47:59.104619 17073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:47:59.104878 17073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:47:59.104921 17073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:47:59.109016 17073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:47:59.109077 17073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:47:59.109166 17073 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260925 04:47:59.114461 17073 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:39863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:43903
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=33783
--tserver_master_addrs=127.0.98.126:33181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17073
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:47:59.115532 17073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:47:59.116541 17073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:47:59.126299 17083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:47:59.126336 17081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:47:59.126323 17080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:47:59.126595 17073 server_base.cc:1034] running on GCE node
I20260925 04:47:59.127018 17073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:47:59.127635 17073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:47:59.128809 17073 hybrid_clock.cc:648] HybridClock initialized: now 1790311679128792 us; error 29 us; skew 500 ppm
Sep 25 04:47:59 dist-test-slave-kghz krb5kdc[15756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311679, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:47:59.131644 17073 init.cc:377] Logged in from keytab as kudu/127.0.98.66@KRBTEST.COM (short username kudu)
I20260925 04:47:59.132519 17073 webserver.cc:460] Webserver started at http://127.0.98.66:33783/ using document root <none> and password file <none>
I20260925 04:47:59.133070 17073 fs_manager.cc:359] Metadata directory not provided
I20260925 04:47:59.133157 17073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:47:59.137105 17073 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 04:47:59.139314 17090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:47:59.140643 17073 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 04:47:59.140782 17073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-1/wal
uuid: "c74e7ff9fd45472cbcc9cb1f76994e3e"
format_stamp: "Formatted at 2026-09-25 04:47:17 on dist-test-slave-kghz"
server_key: "f957305e70af52a37fc92da714a64437"
server_key_iv: "9502451580e9c1b30f9ee1a05a402086"
server_key_version: "encryptionkey@0"
I20260925 04:47:59.141343 17073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:47:59.153790 17073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:47:59.156720 17073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:47:59.156911 17073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:47:59.157477 17073 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:47:59.158299 17073 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:47:59.158365 17073 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:47:59.158444 17073 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:47:59.158489 17073 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:47:59.167906 17073 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:43903
I20260925 04:47:59.167961 17203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:43903 every 8 connection(s)
I20260925 04:47:59.168884 17073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
I20260925 04:47:59.172453 17206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:59.170972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:33181 (local address 127.0.98.66:37211)
0925 04:47:59.171618 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0925 04:47:59.171771 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":108,"threads_started":1}
W20260925 04:47:59.172880 17204 heartbeater.cc:644] Failed to heartbeat to 127.0.98.126:33181 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.126:33181: Client connection negotiation failed: client connection to 127.0.98.126:33181: connect: Connection refused (error 111)
I20260925 04:47:59.173484 17206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:59.173267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:33181 (local address 127.0.98.66:45745)
0925 04:47:59.173381 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0925 04:47:59.173432 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260925 04:47:59.174167 17206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:59.173957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:33181 (local address 127.0.98.66:34367)
0925 04:47:59.174063 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0925 04:47:59.174112 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 04:47:59.174386 17204 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 04:47:59.177815 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 17073
I20260925 04:47:59.178187 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:43709
--local_ip_for_outbound_sockets=127.0.98.67
--tserver_master_addrs=127.0.98.126:33181
--webserver_port=42833
--webserver_interface=127.0.98.67
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:39863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:47:59.289986 17207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:47:59.290378 17207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:47:59.290493 17207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:47:59.294628 17207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:47:59.294724 17207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:47:59.294824 17207 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260925 04:47:59.300207 17207 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:39863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:43709
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=42833
--tserver_master_addrs=127.0.98.126:33181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17207
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:47:59.301337 17207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:47:59.302459 17207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:47:59.312660 17213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:47:59.312652 17212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:47:59.312718 17207 server_base.cc:1034] running on GCE node
W20260925 04:47:59.312664 17215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:47:59.313367 17207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:47:59.314003 17207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:47:59.315171 17207 hybrid_clock.cc:648] HybridClock initialized: now 1790311679315124 us; error 60 us; skew 500 ppm
Sep 25 04:47:59 dist-test-slave-kghz krb5kdc[15756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311679, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:47:59.317900 17207 init.cc:377] Logged in from keytab as kudu/127.0.98.67@KRBTEST.COM (short username kudu)
I20260925 04:47:59.318841 17207 webserver.cc:460] Webserver started at http://127.0.98.67:42833/ using document root <none> and password file <none>
I20260925 04:47:59.319407 17207 fs_manager.cc:359] Metadata directory not provided
I20260925 04:47:59.319495 17207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:47:59.323402 17207 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 04:47:59.325440 17222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:47:59.326648 17207 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 04:47:59.326778 17207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-2/wal
uuid: "0581358747614238835ea146d3d682ea"
format_stamp: "Formatted at 2026-09-25 04:47:17 on dist-test-slave-kghz"
server_key: "d398fe1da8d7289c549debda91682662"
server_key_iv: "288a1f4351930a9ac57305eb1f069db8"
server_key_version: "encryptionkey@0"
I20260925 04:47:59.327337 17207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:47:59.341270 17207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:47:59.344589 17207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:47:59.344794 17207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:47:59.345438 17207 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:47:59.346334 17207 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:47:59.346400 17207 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:47:59.346443 17207 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:47:59.346459 17207 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:47:59.355615 17207 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:43709
I20260925 04:47:59.355700 17335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:43709 every 8 connection(s)
I20260925 04:47:59.356669 17207 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
I20260925 04:47:59.360515 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:59.358964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:33181 (local address 127.0.98.67:44227)
0925 04:47:59.359675 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0925 04:47:59.359796 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":137,"threads_started":1}
W20260925 04:47:59.361039 17336 heartbeater.cc:644] Failed to heartbeat to 127.0.98.126:33181 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.126:33181: Client connection negotiation failed: client connection to 127.0.98.126:33181: connect: Connection refused (error 111)
I20260925 04:47:59.361675 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:59.361454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:33181 (local address 127.0.98.67:44419)
0925 04:47:59.361567 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0925 04:47:59.361617 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 04:47:59.362308 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:47:59.362097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:33181 (local address 127.0.98.67:39117)
0925 04:47:59.362192 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0925 04:47:59.362238 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260925 04:47:59.362471 17336 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 04:47:59.364596 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 17207
I20260925 04:48:00.029577 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:00.029127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:33181 (local address 127.0.98.65:46101)
0925 04:48:00.029401 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:00.029464 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":95,"threads_started":1}
I20260925 04:48:00.175427 17341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:00.175019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:33181 (local address 127.0.98.66:33921)
0925 04:48:00.175268 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:00.175327 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":67,"threads_started":1}
I20260925 04:48:00.363476 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:00.363035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:33181 (local address 127.0.98.67:45969)
0925 04:48:00.363323 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:00.363377 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":68,"threads_started":1}
I20260925 04:48:00.471971 17343 raft_consensus.cc:482] T c1e78a8a6a28410284d1bdaddaf7987f P 7a05a5d169f8498a823f5f8b7ac15f92 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:48:00.472225 17343 raft_consensus.cc:504] T c1e78a8a6a28410284d1bdaddaf7987f P 7a05a5d169f8498a823f5f8b7ac15f92 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c74e7ff9fd45472cbcc9cb1f76994e3e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43903 } } peers { permanent_uuid: "0581358747614238835ea146d3d682ea" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43709 } } peers { permanent_uuid: "7a05a5d169f8498a823f5f8b7ac15f92" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41669 } }
I20260925 04:48:00.473399 17343 leader_election.cc:288] T c1e78a8a6a28410284d1bdaddaf7987f P 7a05a5d169f8498a823f5f8b7ac15f92 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c74e7ff9fd45472cbcc9cb1f76994e3e (127.0.98.66:43903), 0581358747614238835ea146d3d682ea (127.0.98.67:43709)
Sep 25 04:48:00 dist-test-slave-kghz krb5kdc[15756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311678, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for kudu/127.0.98.67@KRBTEST.COM
Sep 25 04:48:00 dist-test-slave-kghz krb5kdc[15756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311678, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for kudu/127.0.98.66@KRBTEST.COM
I20260925 04:48:00.490569 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:00.473922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:53277 (local address 127.0.98.66:43903)
0925 04:48:00.474366 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:00.474386 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:00.474417 (+ 31us) server_negotiation.cc:407] Connection header received
0925 04:48:00.475263 (+ 846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:00.475272 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:00.475733 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:00.476068 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:00.477855 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:00.479447 (+ 1592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:00.481515 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:00.482231 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:00.482445 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:00.486148 (+ 3703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:00.486152 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:00.486164 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:48:00.486195 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:00.487989 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:00.488526 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:00.488534 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:00.488540 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:00.488659 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:00.489010 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:00.489016 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:00.489020 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:00.489421 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:00.489546 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:00.490091 (+ 545us) server_negotiation.cc:299] Negotiation successful
0925 04:48:00.490379 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":92,"threads_started":1}
I20260925 04:48:00.490561 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:00.473641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:43709 (local address 127.0.98.65:38125)
0925 04:48:00.474056 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:00.474072 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:48:00.474170 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:48:00.476186 (+ 2016us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:48:00.476193 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:48:00.476825 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:48:00.477693 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:00.477699 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:00.479634 (+ 1935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:00.479636 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:00.481380 (+ 1744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:00.481391 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:00.482395 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:00.482399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:00.482716 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:00.483568 (+ 852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:48:00.483586 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:48:00.485551 (+ 1965us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:48:00.488033 (+ 2482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:48:00.488041 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:48:00.488057 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:48:00.488429 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:48:00.488797 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:48:00.488799 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:48:00.488801 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:48:00.488921 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:48:00.489564 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:48:00.489567 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:48:00.489794 (+ 227us) client_negotiation.cc:769] Sending connection context
0925 04:48:00.489970 (+ 176us) client_negotiation.cc:240] Negotiation successful
0925 04:48:00.490351 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"mutex_wait_us":380,"spinlock_wait_cycles":43904,"thread_start_us":87,"threads_started":1}
I20260925 04:48:00.490586 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:00.473636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:43903 (local address 127.0.98.65:53277)
0925 04:48:00.473833 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:00.473849 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:48:00.473955 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:48:00.476186 (+ 2231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:48:00.476193 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:48:00.476825 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:48:00.477657 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:00.477679 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:00.479631 (+ 1952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:00.479635 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:00.481380 (+ 1745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:00.481391 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:00.482395 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:00.482399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:00.482716 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:00.483555 (+ 839us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:48:00.483577 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:48:00.485958 (+ 2381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:48:00.488122 (+ 2164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:48:00.488124 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:48:00.488126 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:48:00.488415 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:48:00.488765 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:48:00.488770 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:48:00.488775 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:48:00.488914 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:48:00.489543 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:48:00.489549 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:48:00.489787 (+ 238us) client_negotiation.cc:769] Sending connection context
0925 04:48:00.489969 (+ 182us) client_negotiation.cc:240] Negotiation successful
0925 04:48:00.490427 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":672,"spinlock_wait_cycles":30336}
I20260925 04:48:00.490598 17346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:00.473903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:38125 (local address 127.0.98.67:43709)
0925 04:48:00.474282 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:00.474311 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:00.474356 (+ 45us) server_negotiation.cc:407] Connection header received
0925 04:48:00.475269 (+ 913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:00.475275 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:00.475728 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:00.476068 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:00.477855 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:00.479447 (+ 1592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:00.481512 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:00.482268 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:00.482445 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:00.485855 (+ 3410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:00.485864 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:00.485884 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:48:00.485923 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:00.487890 (+ 1967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:00.488554 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:00.488559 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:00.488563 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:00.488680 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:00.489030 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:00.489033 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:00.489034 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:00.489420 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:00.489559 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:00.490083 (+ 524us) server_negotiation.cc:299] Negotiation successful
0925 04:48:00.490407 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":137,"threads_started":1}
W20260925 04:48:00.493090 16959 leader_election.cc:341] T c1e78a8a6a28410284d1bdaddaf7987f P 7a05a5d169f8498a823f5f8b7ac15f92 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0581358747614238835ea146d3d682ea (127.0.98.67:43709): Not found: Tablet not found: c1e78a8a6a28410284d1bdaddaf7987f
W20260925 04:48:00.493479 16956 leader_election.cc:341] T c1e78a8a6a28410284d1bdaddaf7987f P 7a05a5d169f8498a823f5f8b7ac15f92 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c74e7ff9fd45472cbcc9cb1f76994e3e (127.0.98.66:43903): Not found: Tablet not found: c1e78a8a6a28410284d1bdaddaf7987f
I20260925 04:48:00.493569 16956 leader_election.cc:302] T c1e78a8a6a28410284d1bdaddaf7987f P 7a05a5d169f8498a823f5f8b7ac15f92 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a05a5d169f8498a823f5f8b7ac15f92; no voters: 0581358747614238835ea146d3d682ea, c74e7ff9fd45472cbcc9cb1f76994e3e
I20260925 04:48:00.493852 17343 raft_consensus.cc:2726] T c1e78a8a6a28410284d1bdaddaf7987f P 7a05a5d169f8498a823f5f8b7ac15f92 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 04:48:01.031078 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:01.030585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:33181 (local address 127.0.98.65:33311)
0925 04:48:01.030906 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:01.030982 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":88,"threads_started":1}
I20260925 04:48:01.176558 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:01.176166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:33181 (local address 127.0.98.66:56655)
0925 04:48:01.176416 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:01.176471 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":77,"threads_started":1}
I20260925 04:48:01.364609 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:01.364224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:33181 (local address 127.0.98.67:54099)
0925 04:48:01.364465 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:01.364522 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":80,"threads_started":1}
I20260925 04:48:02.032267 17350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:02.031865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:33181 (local address 127.0.98.65:58053)
0925 04:48:02.032117 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:02.032174 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":86,"threads_started":1}
I20260925 04:48:02.177977 17351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:02.177371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:33181 (local address 127.0.98.66:59259)
0925 04:48:02.177762 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:02.177853 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":79,"threads_started":1}
I20260925 04:48:02.365775 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:02.365338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:33181 (local address 127.0.98.67:33053)
0925 04:48:02.365602 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:02.365667 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":71,"threads_started":1}
I20260925 04:48:02.687408 17353 raft_consensus.cc:482] T c1e78a8a6a28410284d1bdaddaf7987f P 7a05a5d169f8498a823f5f8b7ac15f92 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:48:02.687541 17353 raft_consensus.cc:504] T c1e78a8a6a28410284d1bdaddaf7987f P 7a05a5d169f8498a823f5f8b7ac15f92 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c74e7ff9fd45472cbcc9cb1f76994e3e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43903 } } peers { permanent_uuid: "0581358747614238835ea146d3d682ea" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43709 } } peers { permanent_uuid: "7a05a5d169f8498a823f5f8b7ac15f92" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41669 } }
I20260925 04:48:02.688025 17353 leader_election.cc:288] T c1e78a8a6a28410284d1bdaddaf7987f P 7a05a5d169f8498a823f5f8b7ac15f92 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c74e7ff9fd45472cbcc9cb1f76994e3e (127.0.98.66:43903), 0581358747614238835ea146d3d682ea (127.0.98.67:43709)
W20260925 04:48:02.688944 16959 leader_election.cc:341] T c1e78a8a6a28410284d1bdaddaf7987f P 7a05a5d169f8498a823f5f8b7ac15f92 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0581358747614238835ea146d3d682ea (127.0.98.67:43709): Not found: Tablet not found: c1e78a8a6a28410284d1bdaddaf7987f
W20260925 04:48:02.689093 16956 leader_election.cc:341] T c1e78a8a6a28410284d1bdaddaf7987f P 7a05a5d169f8498a823f5f8b7ac15f92 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c74e7ff9fd45472cbcc9cb1f76994e3e (127.0.98.66:43903): Not found: Tablet not found: c1e78a8a6a28410284d1bdaddaf7987f
I20260925 04:48:02.689169 16956 leader_election.cc:302] T c1e78a8a6a28410284d1bdaddaf7987f P 7a05a5d169f8498a823f5f8b7ac15f92 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a05a5d169f8498a823f5f8b7ac15f92; no voters: 0581358747614238835ea146d3d682ea, c74e7ff9fd45472cbcc9cb1f76994e3e
I20260925 04:48:02.689318 17353 raft_consensus.cc:2726] T c1e78a8a6a28410284d1bdaddaf7987f P 7a05a5d169f8498a823f5f8b7ac15f92 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 04:48:03.033617 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:03.033163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:33181 (local address 127.0.98.65:49147)
0925 04:48:03.033447 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:03.033521 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":89,"threads_started":1}
I20260925 04:48:03.179345 17355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:03.178927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:33181 (local address 127.0.98.66:39783)
0925 04:48:03.179192 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:03.179255 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":112,"threads_started":1}
I20260925 04:48:03.367117 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:03.366710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:33181 (local address 127.0.98.67:41679)
0925 04:48:03.366967 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:03.367023 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":68,"threads_started":1}
I20260925 04:48:04.035362 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:04.034806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:33181 (local address 127.0.98.65:49939)
0925 04:48:04.035162 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:04.035233 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":96,"threads_started":1}
I20260925 04:48:04.180626 17358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:04.180163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:33181 (local address 127.0.98.66:39439)
0925 04:48:04.180447 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:04.180505 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":95,"threads_started":1}
I20260925 04:48:04.368319 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:04.367881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:33181 (local address 127.0.98.67:43795)
0925 04:48:04.368149 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:04.368211 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":76,"threads_started":1}
I20260925 04:48:05.036621 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:05.036183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:33181 (local address 127.0.98.65:60173)
0925 04:48:05.036446 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:05.036505 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":80,"threads_started":1}
I20260925 04:48:05.181912 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:05.181471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:33181 (local address 127.0.98.66:60245)
0925 04:48:05.181753 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:05.181813 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":74,"threads_started":1}
I20260925 04:48:05.369447 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:05.369053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:33181 (local address 127.0.98.67:43665)
0925 04:48:05.369295 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:05.369349 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":71,"threads_started":1}
I20260925 04:48:05.905810 17363 raft_consensus.cc:482] T c1e78a8a6a28410284d1bdaddaf7987f P 7a05a5d169f8498a823f5f8b7ac15f92 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:48:05.905944 17363 raft_consensus.cc:504] T c1e78a8a6a28410284d1bdaddaf7987f P 7a05a5d169f8498a823f5f8b7ac15f92 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c74e7ff9fd45472cbcc9cb1f76994e3e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43903 } } peers { permanent_uuid: "0581358747614238835ea146d3d682ea" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43709 } } peers { permanent_uuid: "7a05a5d169f8498a823f5f8b7ac15f92" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41669 } }
I20260925 04:48:05.906502 17363 leader_election.cc:288] T c1e78a8a6a28410284d1bdaddaf7987f P 7a05a5d169f8498a823f5f8b7ac15f92 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c74e7ff9fd45472cbcc9cb1f76994e3e (127.0.98.66:43903), 0581358747614238835ea146d3d682ea (127.0.98.67:43709)
W20260925 04:48:05.907349 16959 leader_election.cc:341] T c1e78a8a6a28410284d1bdaddaf7987f P 7a05a5d169f8498a823f5f8b7ac15f92 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0581358747614238835ea146d3d682ea (127.0.98.67:43709): Not found: Tablet not found: c1e78a8a6a28410284d1bdaddaf7987f
W20260925 04:48:05.907438 16956 leader_election.cc:341] T c1e78a8a6a28410284d1bdaddaf7987f P 7a05a5d169f8498a823f5f8b7ac15f92 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c74e7ff9fd45472cbcc9cb1f76994e3e (127.0.98.66:43903): Not found: Tablet not found: c1e78a8a6a28410284d1bdaddaf7987f
I20260925 04:48:05.907564 16956 leader_election.cc:302] T c1e78a8a6a28410284d1bdaddaf7987f P 7a05a5d169f8498a823f5f8b7ac15f92 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a05a5d169f8498a823f5f8b7ac15f92; no voters: 0581358747614238835ea146d3d682ea, c74e7ff9fd45472cbcc9cb1f76994e3e
I20260925 04:48:05.907768 17363 raft_consensus.cc:2726] T c1e78a8a6a28410284d1bdaddaf7987f P 7a05a5d169f8498a823f5f8b7ac15f92 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 04:48:06.037896 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:06.037430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:33181 (local address 127.0.98.65:35139)
0925 04:48:06.037736 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:06.037799 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":86,"threads_started":1}
I20260925 04:48:06.183306 17365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:06.182784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:33181 (local address 127.0.98.66:56243)
0925 04:48:06.183117 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:06.183188 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":91,"threads_started":1}
I20260925 04:48:06.370783 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:06.370315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:33181 (local address 127.0.98.67:53659)
0925 04:48:06.370630 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:06.370691 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":92,"threads_started":1}
I20260925 04:48:07.039397 17367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:07.038889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:33181 (local address 127.0.98.65:42449)
0925 04:48:07.039181 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:07.039267 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":96,"threads_started":1}
I20260925 04:48:07.184439 17368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:07.184021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:33181 (local address 127.0.98.66:58817)
0925 04:48:07.184289 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:07.184342 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":79,"threads_started":1}
I20260925 04:48:07.371976 17369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:07.371563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:33181 (local address 127.0.98.67:40205)
0925 04:48:07.371822 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:07.371882 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":73,"threads_started":1}
I20260925 04:48:08.040665 17370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:08.040213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:33181 (local address 127.0.98.65:42033)
0925 04:48:08.040470 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:08.040523 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":83,"threads_started":1}
I20260925 04:48:08.185853 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:08.185393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:33181 (local address 127.0.98.66:40771)
0925 04:48:08.185689 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:08.185754 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":72,"threads_started":1}
I20260925 04:48:08.373241 17372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:08.372787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:33181 (local address 127.0.98.67:43295)
0925 04:48:08.373078 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:08.373143 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":77,"threads_started":1}
I20260925 04:48:09.042016 17373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:09.041614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:33181 (local address 127.0.98.65:54571)
0925 04:48:09.041862 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:09.041920 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":75,"threads_started":1}
I20260925 04:48:09.187330 17374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:09.186782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:33181 (local address 127.0.98.66:47761)
0925 04:48:09.187126 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:09.187210 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":113,"threads_started":1}
I20260925 04:48:09.374742 17375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:09.374230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:33181 (local address 127.0.98.67:43399)
0925 04:48:09.374558 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:09.374614 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":125,"threads_started":1}
I20260925 04:48:10.043293 17376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:10.042891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:33181 (local address 127.0.98.65:56613)
0925 04:48:10.043133 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:10.043195 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":73,"threads_started":1}
I20260925 04:48:10.188555 17377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:10.188140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:33181 (local address 127.0.98.66:54707)
0925 04:48:10.188407 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:10.188460 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":91,"threads_started":1}
I20260925 04:48:10.376083 17378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:10.375519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:33181 (local address 127.0.98.67:48305)
0925 04:48:10.375873 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:10.375936 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":77,"threads_started":1}
I20260925 04:48:10.870010 17379 raft_consensus.cc:482] T c1e78a8a6a28410284d1bdaddaf7987f P 7a05a5d169f8498a823f5f8b7ac15f92 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:48:10.870157 17379 raft_consensus.cc:504] T c1e78a8a6a28410284d1bdaddaf7987f P 7a05a5d169f8498a823f5f8b7ac15f92 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c74e7ff9fd45472cbcc9cb1f76994e3e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43903 } } peers { permanent_uuid: "0581358747614238835ea146d3d682ea" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43709 } } peers { permanent_uuid: "7a05a5d169f8498a823f5f8b7ac15f92" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41669 } }
I20260925 04:48:10.870712 17379 leader_election.cc:288] T c1e78a8a6a28410284d1bdaddaf7987f P 7a05a5d169f8498a823f5f8b7ac15f92 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c74e7ff9fd45472cbcc9cb1f76994e3e (127.0.98.66:43903), 0581358747614238835ea146d3d682ea (127.0.98.67:43709)
W20260925 04:48:10.871511 16956 leader_election.cc:341] T c1e78a8a6a28410284d1bdaddaf7987f P 7a05a5d169f8498a823f5f8b7ac15f92 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c74e7ff9fd45472cbcc9cb1f76994e3e (127.0.98.66:43903): Not found: Tablet not found: c1e78a8a6a28410284d1bdaddaf7987f
W20260925 04:48:10.871606 16959 leader_election.cc:341] T c1e78a8a6a28410284d1bdaddaf7987f P 7a05a5d169f8498a823f5f8b7ac15f92 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0581358747614238835ea146d3d682ea (127.0.98.67:43709): Not found: Tablet not found: c1e78a8a6a28410284d1bdaddaf7987f
I20260925 04:48:10.871722 16959 leader_election.cc:302] T c1e78a8a6a28410284d1bdaddaf7987f P 7a05a5d169f8498a823f5f8b7ac15f92 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a05a5d169f8498a823f5f8b7ac15f92; no voters: 0581358747614238835ea146d3d682ea, c74e7ff9fd45472cbcc9cb1f76994e3e
I20260925 04:48:10.871899 17379 raft_consensus.cc:2726] T c1e78a8a6a28410284d1bdaddaf7987f P 7a05a5d169f8498a823f5f8b7ac15f92 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 04:48:11.044535 17380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:11.044037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:33181 (local address 127.0.98.65:47741)
0925 04:48:11.044307 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:11.044364 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":86,"threads_started":1}
I20260925 04:48:11.189872 17381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:11.189410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:33181 (local address 127.0.98.66:45313)
0925 04:48:11.189682 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:11.189745 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":75,"threads_started":1}
I20260925 04:48:11.377444 17382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:11.376984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:33181 (local address 127.0.98.67:49923)
0925 04:48:11.377277 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:11.377344 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":74,"threads_started":1}
I20260925 04:48:12.045867 17383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:12.045396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:33181 (local address 127.0.98.65:42003)
0925 04:48:12.045665 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:12.045767 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":78,"threads_started":1}
I20260925 04:48:12.191236 17384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:12.190784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:33181 (local address 127.0.98.66:46811)
0925 04:48:12.191073 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:12.191137 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":79,"threads_started":1}
I20260925 04:48:12.378878 17385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:12.378423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:33181 (local address 127.0.98.67:58263)
0925 04:48:12.378679 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:12.378734 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":93,"threads_started":1}
I20260925 04:48:13.047241 17386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:13.046788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:33181 (local address 127.0.98.65:44877)
0925 04:48:13.047077 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:13.047140 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":101,"threads_started":1}
I20260925 04:48:13.192508 17387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:13.192061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:33181 (local address 127.0.98.66:41447)
0925 04:48:13.192325 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:13.192397 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":73,"threads_started":1}
I20260925 04:48:13.380105 17388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:13.379646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:33181 (local address 127.0.98.67:36599)
0925 04:48:13.379941 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:13.380006 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":93,"threads_started":1}
I20260925 04:48:14.048580 17389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:14.048176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:33181 (local address 127.0.98.65:48601)
0925 04:48:14.048431 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:14.048485 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":83,"threads_started":1}
I20260925 04:48:14.193881 17390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:14.193430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:33181 (local address 127.0.98.66:33143)
0925 04:48:14.193688 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:14.193778 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":79,"threads_started":1}
I20260925 04:48:14.373855 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 16900
I20260925 04:48:14.379608 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 17073
I20260925 04:48:14.381672 17391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:14.381286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:33181 (local address 127.0.98.67:52495)
0925 04:48:14.381532 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:14.381583 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":87,"threads_started":1}
I20260925 04:48:14.385047 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 17207
2026-09-25T04:48:14Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59053 ms)
[----------] 30 tests from SecurityITest (244499 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:48:14 dist-test-slave-kghz krb5kdc[17394](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:48:14 dist-test-slave-kghz krb5kdc[17394](info): set up 2 sockets
Sep 25 04:48:14 dist-test-slave-kghz krb5kdc[17394](info): commencing operation
krb5kdc: starting...
W20260925 04:48:15.430073 393 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/krb5kdc/test-user.keytab.
Sep 25 04:48:15 dist-test-slave-kghz krb5kdc[17394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311695, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:48:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:48:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260925 04:48:15.553848 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:46349
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:37449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:46349
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:48:15.664032 17407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:48:15.664351 17407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:48:15.664420 17407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:48:15.668812 17407 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:48:15.668978 17407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:48:15.669049 17407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:48:15.669090 17407 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:48:15.669127 17407 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:48:15.674600 17407 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:37449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:46349
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:46349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17407
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:48:15.675678 17407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:48:15.676692 17407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 04:48:15.685299 17407 server_base.cc:1034] running on GCE node
W20260925 04:48:15.685380 17412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:48:15.685421 17413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:48:15.685282 17415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:48:15.686137 17407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:48:15.687347 17407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:48:15.688540 17407 hybrid_clock.cc:648] HybridClock initialized: now 1790311695688512 us; error 44 us; skew 500 ppm
Sep 25 04:48:15 dist-test-slave-kghz krb5kdc[17394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:48:15.691383 17407 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260925 04:48:15.692272 17407 webserver.cc:460] Webserver started at http://127.0.98.126:33245/ using document root <none> and password file <none>
I20260925 04:48:15.692855 17407 fs_manager.cc:359] Metadata directory not provided
I20260925 04:48:15.692975 17407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:48:15.693215 17407 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:48:15.695393 17407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/minicluster-data/master-0/data/instance:
uuid: "f3b25b4a67c64cc48e125eeb4a3bc1fb"
format_stamp: "Formatted at 2026-09-25 04:48:15 on dist-test-slave-kghz"
server_key: "b69c140f91bf5997cd3aefdc9509c818"
server_key_iv: "34b6d79fd8e4e805d2d9d742d45f861f"
server_key_version: "encryptionkey@0"
I20260925 04:48:15.695984 17407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/minicluster-data/master-0/wal/instance:
uuid: "f3b25b4a67c64cc48e125eeb4a3bc1fb"
format_stamp: "Formatted at 2026-09-25 04:48:15 on dist-test-slave-kghz"
server_key: "b69c140f91bf5997cd3aefdc9509c818"
server_key_iv: "34b6d79fd8e4e805d2d9d742d45f861f"
server_key_version: "encryptionkey@0"
I20260925 04:48:15.699983 17407 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 04:48:15.702554 17422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:48:15.703922 17407 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 04:48:15.704082 17407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/minicluster-data/master-0/wal
uuid: "f3b25b4a67c64cc48e125eeb4a3bc1fb"
format_stamp: "Formatted at 2026-09-25 04:48:15 on dist-test-slave-kghz"
server_key: "b69c140f91bf5997cd3aefdc9509c818"
server_key_iv: "34b6d79fd8e4e805d2d9d742d45f861f"
server_key_version: "encryptionkey@0"
I20260925 04:48:15.704210 17407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:48:15.726362 17407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:48:15.730345 17407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:48:15.730582 17407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:48:15.738521 17407 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:46349
I20260925 04:48:15.738540 17474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:46349 every 8 connection(s)
I20260925 04:48:15.739495 17407 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
I20260925 04:48:15.740182 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 17407
I20260925 04:48:15.740290 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/minicluster-data/master-0/wal/instance
I20260925 04:48:15.740574 393 external_mini_cluster.cc:1428] Setting key 9cb63e25bb9573bde710c5f6bf23e232
I20260925 04:48:15.742648 17475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:48:15.749419 17475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3b25b4a67c64cc48e125eeb4a3bc1fb: Bootstrap starting.
Sep 25 04:48:15 dist-test-slave-kghz krb5kdc[17394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311695, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:48:15.752277 17475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f3b25b4a67c64cc48e125eeb4a3bc1fb: Neither blocks nor log segments found. Creating new log.
I20260925 04:48:15.753175 17475 log.cc:824] T 00000000000000000000000000000000 P f3b25b4a67c64cc48e125eeb4a3bc1fb: Log is configured to *not* fsync() on all Append() calls
I20260925 04:48:15.755523 17475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3b25b4a67c64cc48e125eeb4a3bc1fb: No bootstrap required, opened a new log
I20260925 04:48:15.755599 17478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:15.742060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41798 (local address 127.0.98.126:46349)
0925 04:48:15.742480 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:15.742496 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:15.742531 (+ 35us) server_negotiation.cc:407] Connection header received
0925 04:48:15.743201 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:15.743225 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:15.743550 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:15.743833 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:15.744814 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:15.745844 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:15.747062 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:15.747572 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:15.747694 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:15.750533 (+ 2839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:15.750542 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:15.750560 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:48:15.750601 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:15.752586 (+ 1985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:15.753005 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:15.753011 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:15.753018 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:15.753157 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:15.753477 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:15.753480 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:15.753482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:15.753939 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:15.754146 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:15.754384 (+ 238us) server_negotiation.cc:299] Negotiation successful
0925 04:48:15.754665 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":155,"threads_started":1}
I20260925 04:48:15.759406 17475 raft_consensus.cc:348] T 00000000000000000000000000000000 P f3b25b4a67c64cc48e125eeb4a3bc1fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3b25b4a67c64cc48e125eeb4a3bc1fb" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46349 } }
I20260925 04:48:15.759632 17475 raft_consensus.cc:374] T 00000000000000000000000000000000 P f3b25b4a67c64cc48e125eeb4a3bc1fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:48:15.759670 17475 raft_consensus.cc:729] T 00000000000000000000000000000000 P f3b25b4a67c64cc48e125eeb4a3bc1fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3b25b4a67c64cc48e125eeb4a3bc1fb, State: Initialized, Role: FOLLOWER
I20260925 04:48:15.760046 17475 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3b25b4a67c64cc48e125eeb4a3bc1fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3b25b4a67c64cc48e125eeb4a3bc1fb" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46349 } }
I20260925 04:48:15.760159 17475 raft_consensus.cc:388] T 00000000000000000000000000000000 P f3b25b4a67c64cc48e125eeb4a3bc1fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:48:15.760197 17475 raft_consensus.cc:482] T 00000000000000000000000000000000 P f3b25b4a67c64cc48e125eeb4a3bc1fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:48:15.760269 17475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f3b25b4a67c64cc48e125eeb4a3bc1fb [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:48:15.761132 17475 raft_consensus.cc:504] T 00000000000000000000000000000000 P f3b25b4a67c64cc48e125eeb4a3bc1fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3b25b4a67c64cc48e125eeb4a3bc1fb" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46349 } }
I20260925 04:48:15.761423 17475 leader_election.cc:302] T 00000000000000000000000000000000 P f3b25b4a67c64cc48e125eeb4a3bc1fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f3b25b4a67c64cc48e125eeb4a3bc1fb; no voters:
I20260925 04:48:15.761677 17475 leader_election.cc:288] T 00000000000000000000000000000000 P f3b25b4a67c64cc48e125eeb4a3bc1fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:48:15.761824 17480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f3b25b4a67c64cc48e125eeb4a3bc1fb [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:48:15.762135 17480 raft_consensus.cc:686] T 00000000000000000000000000000000 P f3b25b4a67c64cc48e125eeb4a3bc1fb [term 1 LEADER]: Becoming Leader. State: Replica: f3b25b4a67c64cc48e125eeb4a3bc1fb, State: Running, Role: LEADER
I20260925 04:48:15.762458 17480 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3b25b4a67c64cc48e125eeb4a3bc1fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3b25b4a67c64cc48e125eeb4a3bc1fb" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46349 } }
I20260925 04:48:15.763021 17475 sys_catalog.cc:564] T 00000000000000000000000000000000 P f3b25b4a67c64cc48e125eeb4a3bc1fb [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:48:15.764019 17481 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3b25b4a67c64cc48e125eeb4a3bc1fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f3b25b4a67c64cc48e125eeb4a3bc1fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3b25b4a67c64cc48e125eeb4a3bc1fb" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46349 } } }
I20260925 04:48:15.764150 17481 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3b25b4a67c64cc48e125eeb4a3bc1fb [sys.catalog]: This master's current role is: LEADER
I20260925 04:48:15.764555 17489 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:48:15.764503 17482 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3b25b4a67c64cc48e125eeb4a3bc1fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3b25b4a67c64cc48e125eeb4a3bc1fb. Latest consensus state: current_term: 1 leader_uuid: "f3b25b4a67c64cc48e125eeb4a3bc1fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3b25b4a67c64cc48e125eeb4a3bc1fb" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46349 } } }
I20260925 04:48:15.764595 17482 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3b25b4a67c64cc48e125eeb4a3bc1fb [sys.catalog]: This master's current role is: LEADER
I20260925 04:48:15.767267 17489 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:48:15.772878 17489 catalog_manager.cc:1347] Generated new cluster ID: 3443991527554d58a127f0af8e213d10
I20260925 04:48:15.772945 17489 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:48:15.792502 17489 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:48:15.793784 17489 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:48:15.798899 17489 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f3b25b4a67c64cc48e125eeb4a3bc1fb: Generated new TSK 0
I20260925 04:48:15.799566 17489 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/krb5kdc/kudu_127.0.98.65.keytab.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/krb5kdc/kudu_127.0.98.65.keytab.
I20260925 04:48:15.829028 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:0
--local_ip_for_outbound_sockets=127.0.98.65
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:46349
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:37449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:48:15.944828 17501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:48:15.945142 17501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:48:15.945217 17501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:48:15.949379 17501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:48:15.949482 17501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:48:15.949582 17501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260925 04:48:15.955148 17501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:37449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:46349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17501
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:48:15.956302 17501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:48:15.957407 17501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:48:15.967983 17509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:48:15.968041 17506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:48:15.967983 17507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:48:15.968328 17501 server_base.cc:1034] running on GCE node
I20260925 04:48:15.968768 17501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:48:15.969403 17501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:48:15.970634 17501 hybrid_clock.cc:648] HybridClock initialized: now 1790311695970618 us; error 32 us; skew 500 ppm
Sep 25 04:48:15 dist-test-slave-kghz krb5kdc[17394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:48:15.973630 17501 init.cc:377] Logged in from keytab as kudu/127.0.98.65@KRBTEST.COM (short username kudu)
I20260925 04:48:15.974609 17501 webserver.cc:460] Webserver started at http://127.0.98.65:35993/ using document root <none> and password file <none>
I20260925 04:48:15.975201 17501 fs_manager.cc:359] Metadata directory not provided
I20260925 04:48:15.975296 17501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:48:15.975513 17501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:48:15.977538 17501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/minicluster-data/ts-0/data/instance:
uuid: "99144cb9a1414f9b80ae8a70887961cb"
format_stamp: "Formatted at 2026-09-25 04:48:15 on dist-test-slave-kghz"
server_key: "ac3a44a9432f27abf6adc1df18b2080c"
server_key_iv: "610b473049f4c3539fe7a1a34d520d1c"
server_key_version: "encryptionkey@0"
I20260925 04:48:15.978120 17501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/minicluster-data/ts-0/wal/instance:
uuid: "99144cb9a1414f9b80ae8a70887961cb"
format_stamp: "Formatted at 2026-09-25 04:48:15 on dist-test-slave-kghz"
server_key: "ac3a44a9432f27abf6adc1df18b2080c"
server_key_iv: "610b473049f4c3539fe7a1a34d520d1c"
server_key_version: "encryptionkey@0"
I20260925 04:48:15.981842 17501 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260925 04:48:15.984485 17516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:48:15.986129 17501 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 04:48:15.986238 17501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/minicluster-data/ts-0/wal
uuid: "99144cb9a1414f9b80ae8a70887961cb"
format_stamp: "Formatted at 2026-09-25 04:48:15 on dist-test-slave-kghz"
server_key: "ac3a44a9432f27abf6adc1df18b2080c"
server_key_iv: "610b473049f4c3539fe7a1a34d520d1c"
server_key_version: "encryptionkey@0"
I20260925 04:48:15.986402 17501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:48:15.998973 17501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:48:16.001888 17501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:48:16.002044 17501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:48:16.002640 17501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:48:16.003424 17501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:48:16.003470 17501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:48:16.003510 17501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:48:16.003525 17501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:48:16.012821 17501 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:44863
I20260925 04:48:16.012838 17629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:44863 every 8 connection(s)
I20260925 04:48:16.013737 17501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
I20260925 04:48:16.015807 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 17501
I20260925 04:48:16.015973 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/minicluster-data/ts-0/wal/instance
I20260925 04:48:16.016263 393 external_mini_cluster.cc:1428] Setting key 86106e8369050d81dc87ebf532982226
Sep 25 04:48:16 dist-test-slave-kghz krb5kdc[17394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.66@KRBTEST.COM" created.
I20260925 04:48:16.029572 17478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:16.015671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:40501 (local address 127.0.98.126:46349)
0925 04:48:16.016770 (+ 1099us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:16.016778 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:16.016799 (+ 21us) server_negotiation.cc:407] Connection header received
0925 04:48:16.018201 (+ 1402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:16.018204 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:16.018283 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:16.018373 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:16.019547 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:16.020087 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:16.021905 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:16.022142 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:16.022310 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:16.025222 (+ 2912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:16.025225 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:16.025227 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:48:16.025252 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:16.026884 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:16.027479 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:16.027482 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:16.027483 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:16.027561 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:16.028049 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:16.028052 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:16.028053 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:16.028239 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:16.028320 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:16.029130 (+ 810us) server_negotiation.cc:299] Negotiation successful
0925 04:48:16.029286 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":990}
I20260925 04:48:16.030239 17632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:16.016114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:46349 (local address 127.0.98.65:40501)
0925 04:48:16.016590 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:16.016643 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 04:48:16.017987 (+ 1344us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:48:16.018559 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:48:16.018566 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:48:16.018923 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:48:16.019397 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:16.019410 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:16.020216 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:16.020222 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:16.021734 (+ 1512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:16.021747 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:16.022344 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:16.022349 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:16.022584 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:16.023252 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:48:16.023280 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:48:16.025036 (+ 1756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:48:16.027055 (+ 2019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:48:16.027060 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:48:16.027075 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:48:16.027381 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:48:16.027686 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:48:16.027689 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:48:16.027690 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:48:16.027944 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:48:16.028421 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:48:16.028427 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:48:16.028846 (+ 419us) client_negotiation.cc:769] Sending connection context
0925 04:48:16.029052 (+ 206us) client_negotiation.cc:240] Negotiation successful
0925 04:48:16.029334 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":132,"threads_started":1}
I20260925 04:48:16.031746 17630 heartbeater.cc:347] Connected to a master server at 127.0.98.126:46349
I20260925 04:48:16.032015 17630 heartbeater.cc:464] Registering TS with master...
I20260925 04:48:16.032524 17630 heartbeater.cc:511] Master 127.0.98.126:46349 requested a full tablet report, sending...
I20260925 04:48:16.034133 17439 ts_manager.cc:194] Registered new tserver with Master: 99144cb9a1414f9b80ae8a70887961cb (127.0.98.65:44863)
I20260925 04:48:16.035293 17439 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.65@KRBTEST.COM'} at 127.0.98.65:40501
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/krb5kdc/kudu_127.0.98.66.keytab.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/krb5kdc/kudu_127.0.98.66.keytab.
I20260925 04:48:16.044899 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:0
--local_ip_for_outbound_sockets=127.0.98.66
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:46349
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:37449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:48:16.155174 17635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:48:16.155491 17635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:48:16.155565 17635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:48:16.159621 17635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:48:16.159729 17635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:48:16.159838 17635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260925 04:48:16.165568 17635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:37449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:46349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17635
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:48:16.166698 17635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:48:16.167695 17635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:48:16.177476 17643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:48:16.177511 17640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:48:16.177611 17635 server_base.cc:1034] running on GCE node
W20260925 04:48:16.177541 17641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:48:16.178143 17635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:48:16.178826 17635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:48:16.180022 17635 hybrid_clock.cc:648] HybridClock initialized: now 1790311696180007 us; error 28 us; skew 500 ppm
Sep 25 04:48:16 dist-test-slave-kghz krb5kdc[17394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311696, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:48:16.182715 17635 init.cc:377] Logged in from keytab as kudu/127.0.98.66@KRBTEST.COM (short username kudu)
I20260925 04:48:16.183670 17635 webserver.cc:460] Webserver started at http://127.0.98.66:40309/ using document root <none> and password file <none>
I20260925 04:48:16.184237 17635 fs_manager.cc:359] Metadata directory not provided
I20260925 04:48:16.184329 17635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:48:16.184568 17635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:48:16.186646 17635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/minicluster-data/ts-1/data/instance:
uuid: "d822421a171a4b71941784569da5a41a"
format_stamp: "Formatted at 2026-09-25 04:48:16 on dist-test-slave-kghz"
server_key: "062b32deee48e97c4f5cac64304a05ba"
server_key_iv: "db58748e7e16d752f0a0759f3f3e162d"
server_key_version: "encryptionkey@0"
I20260925 04:48:16.187217 17635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/minicluster-data/ts-1/wal/instance:
uuid: "d822421a171a4b71941784569da5a41a"
format_stamp: "Formatted at 2026-09-25 04:48:16 on dist-test-slave-kghz"
server_key: "062b32deee48e97c4f5cac64304a05ba"
server_key_iv: "db58748e7e16d752f0a0759f3f3e162d"
server_key_version: "encryptionkey@0"
I20260925 04:48:16.191177 17635 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 04:48:16.193589 17650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:48:16.194877 17635 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 04:48:16.195015 17635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/minicluster-data/ts-1/wal
uuid: "d822421a171a4b71941784569da5a41a"
format_stamp: "Formatted at 2026-09-25 04:48:16 on dist-test-slave-kghz"
server_key: "062b32deee48e97c4f5cac64304a05ba"
server_key_iv: "db58748e7e16d752f0a0759f3f3e162d"
server_key_version: "encryptionkey@0"
I20260925 04:48:16.195129 17635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:48:16.213543 17635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:48:16.216962 17635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:48:16.217183 17635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:48:16.217818 17635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:48:16.218724 17635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:48:16.218796 17635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:48:16.218863 17635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:48:16.218916 17635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:48:16.228060 17635 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:36143
I20260925 04:48:16.228097 17763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:36143 every 8 connection(s)
I20260925 04:48:16.229024 17635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
I20260925 04:48:16.231336 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 17635
I20260925 04:48:16.231465 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/minicluster-data/ts-1/wal/instance
I20260925 04:48:16.231803 393 external_mini_cluster.cc:1428] Setting key 2c0118f4c462c3566576864e1a602f90
Sep 25 04:48:16 dist-test-slave-kghz krb5kdc[17394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311696, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.67@KRBTEST.COM; defaulting to no policy
I20260925 04:48:16.243585 17478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:16.230981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:39177 (local address 127.0.98.126:46349)
0925 04:48:16.231142 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:16.231146 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:16.232212 (+ 1066us) server_negotiation.cc:407] Connection header received
0925 04:48:16.233612 (+ 1400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:16.233615 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:16.233671 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:16.233745 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:16.235006 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:16.235549 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:16.236695 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:16.236933 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:16.237046 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:16.239577 (+ 2531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:16.239580 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:16.239583 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:48:16.239616 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:16.241317 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:16.241859 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:16.241864 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:16.241865 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:16.241980 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:16.242381 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:16.242384 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:16.242386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:16.242587 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:16.242702 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:16.243273 (+ 571us) server_negotiation.cc:299] Negotiation successful
0925 04:48:16.243410 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.67@KRBTEST.COM" created.
I20260925 04:48:16.244496 17766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:16.231438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:46349 (local address 127.0.98.66:39177)
0925 04:48:16.232045 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:16.232089 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 04:48:16.233398 (+ 1309us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:48:16.233935 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:48:16.233942 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:48:16.234378 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:48:16.234864 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:16.234877 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:16.235686 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:16.235692 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:16.236568 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:16.236577 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:16.237080 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:16.237084 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:16.237256 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:16.237882 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:48:16.237899 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:48:16.239416 (+ 1517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:48:16.241461 (+ 2045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:48:16.241468 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:48:16.241484 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:48:16.241749 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:48:16.242085 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:48:16.242088 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:48:16.242090 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:48:16.242240 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:48:16.242703 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:48:16.242711 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:48:16.242971 (+ 260us) client_negotiation.cc:769] Sending connection context
0925 04:48:16.243182 (+ 211us) client_negotiation.cc:240] Negotiation successful
0925 04:48:16.243530 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":128,"threads_started":1}
I20260925 04:48:16.245704 17764 heartbeater.cc:347] Connected to a master server at 127.0.98.126:46349
I20260925 04:48:16.245973 17764 heartbeater.cc:464] Registering TS with master...
I20260925 04:48:16.246558 17764 heartbeater.cc:511] Master 127.0.98.126:46349 requested a full tablet report, sending...
I20260925 04:48:16.247718 17439 ts_manager.cc:194] Registered new tserver with Master: d822421a171a4b71941784569da5a41a (127.0.98.66:36143)
I20260925 04:48:16.248605 17439 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.66@KRBTEST.COM'} at 127.0.98.66:39177
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/krb5kdc/kudu_127.0.98.67.keytab.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/krb5kdc/kudu_127.0.98.67.keytab.
I20260925 04:48:16.261817 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:0
--local_ip_for_outbound_sockets=127.0.98.67
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:46349
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:37449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:48:16.376065 17769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:48:16.376354 17769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:48:16.376402 17769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:48:16.380576 17769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:48:16.380653 17769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:48:16.380717 17769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260925 04:48:16.386679 17769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:37449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:46349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17769
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:48:16.387853 17769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:48:16.388876 17769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:48:16.398873 17775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:48:16.398873 17774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:48:16.399128 17769 server_base.cc:1034] running on GCE node
W20260925 04:48:16.399336 17777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:48:16.399799 17769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:48:16.400480 17769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:48:16.401676 17769 hybrid_clock.cc:648] HybridClock initialized: now 1790311696401659 us; error 30 us; skew 500 ppm
Sep 25 04:48:16 dist-test-slave-kghz krb5kdc[17394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311696, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:48:16.404634 17769 init.cc:377] Logged in from keytab as kudu/127.0.98.67@KRBTEST.COM (short username kudu)
I20260925 04:48:16.405685 17769 webserver.cc:460] Webserver started at http://127.0.98.67:32795/ using document root <none> and password file <none>
I20260925 04:48:16.406347 17769 fs_manager.cc:359] Metadata directory not provided
I20260925 04:48:16.406438 17769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:48:16.406689 17769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:48:16.408927 17769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/minicluster-data/ts-2/data/instance:
uuid: "4183ce246ebd4981ab49d5a023bc9bef"
format_stamp: "Formatted at 2026-09-25 04:48:16 on dist-test-slave-kghz"
server_key: "67a81b820132cbdc327378b00b15a432"
server_key_iv: "22e0e1a1f8d0ae5e540d31c0d5c1fbea"
server_key_version: "encryptionkey@0"
I20260925 04:48:16.409507 17769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/minicluster-data/ts-2/wal/instance:
uuid: "4183ce246ebd4981ab49d5a023bc9bef"
format_stamp: "Formatted at 2026-09-25 04:48:16 on dist-test-slave-kghz"
server_key: "67a81b820132cbdc327378b00b15a432"
server_key_iv: "22e0e1a1f8d0ae5e540d31c0d5c1fbea"
server_key_version: "encryptionkey@0"
I20260925 04:48:16.413293 17769 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260925 04:48:16.415838 17784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:48:16.417311 17769 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 04:48:16.417467 17769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/minicluster-data/ts-2/wal
uuid: "4183ce246ebd4981ab49d5a023bc9bef"
format_stamp: "Formatted at 2026-09-25 04:48:16 on dist-test-slave-kghz"
server_key: "67a81b820132cbdc327378b00b15a432"
server_key_iv: "22e0e1a1f8d0ae5e540d31c0d5c1fbea"
server_key_version: "encryptionkey@0"
I20260925 04:48:16.417593 17769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:48:16.441746 17769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:48:16.445405 17769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:48:16.445637 17769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:48:16.446346 17769 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:48:16.447266 17769 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:48:16.447341 17769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:48:16.447412 17769 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:48:16.447456 17769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:48:16.456864 17769 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:43279
I20260925 04:48:16.456908 17897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:43279 every 8 connection(s)
I20260925 04:48:16.457858 17769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
I20260925 04:48:16.458881 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 17769
I20260925 04:48:16.459017 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/minicluster-data/ts-2/wal/instance
I20260925 04:48:16.459299 393 external_mini_cluster.cc:1428] Setting key 4d8231a82b18e1f61859529a213f8e18
Sep 25 04:48:16 dist-test-slave-kghz krb5kdc[17394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311696, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:48:16.475666 17478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:16.459967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:52613 (local address 127.0.98.126:46349)
0925 04:48:16.460137 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:16.460142 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:16.461848 (+ 1706us) server_negotiation.cc:407] Connection header received
0925 04:48:16.463888 (+ 2040us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:16.463891 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:16.463980 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:16.464076 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:16.465853 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:16.466927 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:16.468486 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:16.468922 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:16.469083 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:16.471703 (+ 2620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:16.471706 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:16.471708 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:48:16.471736 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:16.473418 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:16.474085 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:16.474088 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:16.474090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:16.474188 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:16.474604 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:16.474606 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:16.474607 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:16.474788 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:16.474899 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:16.475442 (+ 543us) server_negotiation.cc:299] Negotiation successful
0925 04:48:16.475547 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 04:48:16.476564 17900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:16.460459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:46349 (local address 127.0.98.67:52613)
0925 04:48:16.461676 (+ 1217us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:16.461735 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 04:48:16.463578 (+ 1843us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:48:16.464363 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:48:16.464376 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:48:16.464942 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:48:16.465680 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:16.465702 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:16.467097 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:16.467103 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:16.468374 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:16.468385 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:16.469147 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:16.469152 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:16.469366 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:16.469936 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:48:16.469953 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:48:16.471525 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:48:16.473574 (+ 2049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:48:16.473580 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:48:16.473598 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:48:16.473965 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:48:16.474348 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:48:16.474351 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:48:16.474354 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:48:16.474516 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:48:16.474907 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:48:16.474912 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:48:16.475171 (+ 259us) client_negotiation.cc:769] Sending connection context
0925 04:48:16.475349 (+ 178us) client_negotiation.cc:240] Negotiation successful
0925 04:48:16.475647 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":905,"thread_start_us":142,"threads_started":1}
I20260925 04:48:16.477689 17898 heartbeater.cc:347] Connected to a master server at 127.0.98.126:46349
I20260925 04:48:16.477941 17898 heartbeater.cc:464] Registering TS with master...
I20260925 04:48:16.478509 17898 heartbeater.cc:511] Master 127.0.98.126:46349 requested a full tablet report, sending...
I20260925 04:48:16.479555 17439 ts_manager.cc:194] Registered new tserver with Master: 4183ce246ebd4981ab49d5a023bc9bef (127.0.98.67:43279)
I20260925 04:48:16.480195 17439 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.67@KRBTEST.COM'} at 127.0.98.67:52613
I20260925 04:48:16.486065 393 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 04:48:16.496577 17478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:16.488241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41802 (local address 127.0.98.126:46349)
0925 04:48:16.488403 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:16.488409 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:16.488614 (+ 205us) server_negotiation.cc:407] Connection header received
0925 04:48:16.488788 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:16.488792 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:16.488850 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:16.488976 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:16.489756 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:16.490468 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:16.491429 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:16.491674 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:16.491794 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:16.493247 (+ 1453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:16.493251 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:16.493254 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:48:16.493286 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:16.494685 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:16.495154 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:16.495159 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:16.495161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:16.495250 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:16.495567 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:16.495571 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:16.495573 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:16.495817 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:16.495982 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:16.496249 (+ 267us) server_negotiation.cc:299] Negotiation successful
0925 04:48:16.496390 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 04:48:16.499869 17439 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41802:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:48:16.503028 17439 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:48:16.519061 17912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:16.512533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:36143 (local address 127.0.0.1:58054)
0925 04:48:16.513231 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:16.513249 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 04:48:16.513390 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:48:16.514517 (+ 1127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:48:16.514520 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:48:16.514542 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:48:16.514660 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:16.514667 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:16.516137 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:16.516143 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:16.517346 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:16.517356 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:16.518443 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:16.518447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:16.518573 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:16.518624 (+ 51us) client_negotiation.cc:769] Sending connection context
0925 04:48:16.518742 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 04:48:16.518879 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"mutex_wait_us":12,"spinlock_wait_cycles":1920,"thread_start_us":134,"threads_started":1}
I20260925 04:48:16.519373 17911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:16.512500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33674 (local address 127.0.98.67:43279)
0925 04:48:16.513893 (+ 1393us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:16.513906 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:16.513927 (+ 21us) server_negotiation.cc:407] Connection header received
0925 04:48:16.513995 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:16.514000 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:16.514168 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:16.514403 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:48:16.514803 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:16.515630 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:16.517834 (+ 2204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:16.518620 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:16.518751 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:16.518911 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:16.518972 (+ 61us) server_negotiation.cc:299] Negotiation successful
0925 04:48:16.519074 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1288,"thread_start_us":127,"threads_started":1}
I20260925 04:48:16.519520 17909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:16.512766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:44863 (local address 127.0.0.1:41644)
0925 04:48:16.513609 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:16.513622 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:48:16.513721 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:48:16.514280 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:48:16.514295 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:48:16.514319 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:48:16.514496 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:16.514504 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:16.516002 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:16.516005 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:16.517674 (+ 1669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:16.517686 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:16.518970 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:16.518973 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:16.519057 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:16.519080 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 04:48:16.519148 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 04:48:16.519415 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":778,"mutex_wait_us":209,"thread_start_us":112,"threads_started":1}
I20260925 04:48:16.519634 17913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:16.513033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58054 (local address 127.0.98.66:36143)
0925 04:48:16.514009 (+ 976us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:16.514023 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:16.514046 (+ 23us) server_negotiation.cc:407] Connection header received
0925 04:48:16.514121 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:16.514126 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:16.514382 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:16.514542 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:48:16.514886 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:16.515990 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:16.517475 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:16.518306 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:16.518425 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:16.518575 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:16.519318 (+ 743us) server_negotiation.cc:299] Negotiation successful
0925 04:48:16.519458 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":109,"threads_started":1}
I20260925 04:48:16.520087 17914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:16.513165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41644 (local address 127.0.98.65:44863)
0925 04:48:16.513586 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:16.513602 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:16.513731 (+ 129us) server_negotiation.cc:407] Connection header received
0925 04:48:16.513824 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:16.513830 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:16.514075 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:16.514284 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:48:16.514634 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:16.515867 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:16.517815 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:16.518633 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:16.518772 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:16.518911 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:16.519828 (+ 917us) server_negotiation.cc:299] Negotiation successful
0925 04:48:16.519940 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":98,"threads_started":1}
I20260925 04:48:16.521210 17910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:16.512326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:43279 (local address 127.0.0.1:33674)
0925 04:48:16.513182 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:16.513237 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 04:48:16.513381 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:48:16.514563 (+ 1182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:48:16.514566 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:48:16.514586 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:48:16.514666 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:16.514671 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:16.515779 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:16.515787 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:16.517674 (+ 1887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:16.517686 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:16.518765 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:16.518769 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:16.518857 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:16.518879 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 04:48:16.518946 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 04:48:16.519017 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"mutex_wait_us":202,"spinlock_wait_cycles":5760,"thread_start_us":95,"threads_started":1}
I20260925 04:48:16.523393 17564 tablet_service.cc:1467] Processing CreateTablet for tablet e759703c32a344b08da7bd8a966365ea (DEFAULT_TABLE table=test-table [id=c5c95151cc0c43acb6b5f44fab89abb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:48:16.524120 17832 tablet_service.cc:1467] Processing CreateTablet for tablet e759703c32a344b08da7bd8a966365ea (DEFAULT_TABLE table=test-table [id=c5c95151cc0c43acb6b5f44fab89abb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:48:16.524641 17564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e759703c32a344b08da7bd8a966365ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:48:16.524770 17698 tablet_service.cc:1467] Processing CreateTablet for tablet e759703c32a344b08da7bd8a966365ea (DEFAULT_TABLE table=test-table [id=c5c95151cc0c43acb6b5f44fab89abb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:48:16.525187 17832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e759703c32a344b08da7bd8a966365ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:48:16.525782 17698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e759703c32a344b08da7bd8a966365ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:48:16.530953 17915 tablet_bootstrap.cc:492] T e759703c32a344b08da7bd8a966365ea P 99144cb9a1414f9b80ae8a70887961cb: Bootstrap starting.
I20260925 04:48:16.532986 17916 tablet_bootstrap.cc:492] T e759703c32a344b08da7bd8a966365ea P d822421a171a4b71941784569da5a41a: Bootstrap starting.
I20260925 04:48:16.532986 17917 tablet_bootstrap.cc:492] T e759703c32a344b08da7bd8a966365ea P 4183ce246ebd4981ab49d5a023bc9bef: Bootstrap starting.
I20260925 04:48:16.533402 17915 tablet_bootstrap.cc:654] T e759703c32a344b08da7bd8a966365ea P 99144cb9a1414f9b80ae8a70887961cb: Neither blocks nor log segments found. Creating new log.
I20260925 04:48:16.534128 17915 log.cc:824] T e759703c32a344b08da7bd8a966365ea P 99144cb9a1414f9b80ae8a70887961cb: Log is configured to *not* fsync() on all Append() calls
I20260925 04:48:16.535246 17917 tablet_bootstrap.cc:654] T e759703c32a344b08da7bd8a966365ea P 4183ce246ebd4981ab49d5a023bc9bef: Neither blocks nor log segments found. Creating new log.
I20260925 04:48:16.535692 17916 tablet_bootstrap.cc:654] T e759703c32a344b08da7bd8a966365ea P d822421a171a4b71941784569da5a41a: Neither blocks nor log segments found. Creating new log.
I20260925 04:48:16.536247 17915 tablet_bootstrap.cc:492] T e759703c32a344b08da7bd8a966365ea P 99144cb9a1414f9b80ae8a70887961cb: No bootstrap required, opened a new log
I20260925 04:48:16.536459 17915 ts_tablet_manager.cc:1397] T e759703c32a344b08da7bd8a966365ea P 99144cb9a1414f9b80ae8a70887961cb: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 04:48:16.536520 17917 log.cc:824] T e759703c32a344b08da7bd8a966365ea P 4183ce246ebd4981ab49d5a023bc9bef: Log is configured to *not* fsync() on all Append() calls
I20260925 04:48:16.536520 17916 log.cc:824] T e759703c32a344b08da7bd8a966365ea P d822421a171a4b71941784569da5a41a: Log is configured to *not* fsync() on all Append() calls
I20260925 04:48:16.538686 17916 tablet_bootstrap.cc:492] T e759703c32a344b08da7bd8a966365ea P d822421a171a4b71941784569da5a41a: No bootstrap required, opened a new log
I20260925 04:48:16.538833 17917 tablet_bootstrap.cc:492] T e759703c32a344b08da7bd8a966365ea P 4183ce246ebd4981ab49d5a023bc9bef: No bootstrap required, opened a new log
I20260925 04:48:16.538884 17916 ts_tablet_manager.cc:1397] T e759703c32a344b08da7bd8a966365ea P d822421a171a4b71941784569da5a41a: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 04:48:16.538980 17917 ts_tablet_manager.cc:1397] T e759703c32a344b08da7bd8a966365ea P 4183ce246ebd4981ab49d5a023bc9bef: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 04:48:16.540758 17915 raft_consensus.cc:348] T e759703c32a344b08da7bd8a966365ea P 99144cb9a1414f9b80ae8a70887961cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4183ce246ebd4981ab49d5a023bc9bef" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43279 } } peers { permanent_uuid: "d822421a171a4b71941784569da5a41a" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 36143 } } peers { permanent_uuid: "99144cb9a1414f9b80ae8a70887961cb" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 44863 } }
I20260925 04:48:16.541031 17915 raft_consensus.cc:374] T e759703c32a344b08da7bd8a966365ea P 99144cb9a1414f9b80ae8a70887961cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:48:16.541085 17915 raft_consensus.cc:729] T e759703c32a344b08da7bd8a966365ea P 99144cb9a1414f9b80ae8a70887961cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99144cb9a1414f9b80ae8a70887961cb, State: Initialized, Role: FOLLOWER
I20260925 04:48:16.541587 17915 consensus_queue.cc:260] T e759703c32a344b08da7bd8a966365ea P 99144cb9a1414f9b80ae8a70887961cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4183ce246ebd4981ab49d5a023bc9bef" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43279 } } peers { permanent_uuid: "d822421a171a4b71941784569da5a41a" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 36143 } } peers { permanent_uuid: "99144cb9a1414f9b80ae8a70887961cb" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 44863 } }
I20260925 04:48:16.542300 17630 heartbeater.cc:503] Master 127.0.98.126:46349 was elected leader, sending a full tablet report...
I20260925 04:48:16.542688 17915 ts_tablet_manager.cc:1428] T e759703c32a344b08da7bd8a966365ea P 99144cb9a1414f9b80ae8a70887961cb: Time spent starting tablet: real 0.006s user 0.002s sys 0.005s
I20260925 04:48:16.542852 17916 raft_consensus.cc:348] T e759703c32a344b08da7bd8a966365ea P d822421a171a4b71941784569da5a41a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4183ce246ebd4981ab49d5a023bc9bef" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43279 } } peers { permanent_uuid: "d822421a171a4b71941784569da5a41a" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 36143 } } peers { permanent_uuid: "99144cb9a1414f9b80ae8a70887961cb" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 44863 } }
I20260925 04:48:16.542852 17917 raft_consensus.cc:348] T e759703c32a344b08da7bd8a966365ea P 4183ce246ebd4981ab49d5a023bc9bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4183ce246ebd4981ab49d5a023bc9bef" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43279 } } peers { permanent_uuid: "d822421a171a4b71941784569da5a41a" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 36143 } } peers { permanent_uuid: "99144cb9a1414f9b80ae8a70887961cb" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 44863 } }
I20260925 04:48:16.543133 17916 raft_consensus.cc:374] T e759703c32a344b08da7bd8a966365ea P d822421a171a4b71941784569da5a41a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:48:16.543133 17917 raft_consensus.cc:374] T e759703c32a344b08da7bd8a966365ea P 4183ce246ebd4981ab49d5a023bc9bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:48:16.543196 17917 raft_consensus.cc:729] T e759703c32a344b08da7bd8a966365ea P 4183ce246ebd4981ab49d5a023bc9bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4183ce246ebd4981ab49d5a023bc9bef, State: Initialized, Role: FOLLOWER
I20260925 04:48:16.543205 17916 raft_consensus.cc:729] T e759703c32a344b08da7bd8a966365ea P d822421a171a4b71941784569da5a41a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d822421a171a4b71941784569da5a41a, State: Initialized, Role: FOLLOWER
I20260925 04:48:16.543638 17916 consensus_queue.cc:260] T e759703c32a344b08da7bd8a966365ea P d822421a171a4b71941784569da5a41a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4183ce246ebd4981ab49d5a023bc9bef" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43279 } } peers { permanent_uuid: "d822421a171a4b71941784569da5a41a" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 36143 } } peers { permanent_uuid: "99144cb9a1414f9b80ae8a70887961cb" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 44863 } }
I20260925 04:48:16.543748 17917 consensus_queue.cc:260] T e759703c32a344b08da7bd8a966365ea P 4183ce246ebd4981ab49d5a023bc9bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4183ce246ebd4981ab49d5a023bc9bef" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43279 } } peers { permanent_uuid: "d822421a171a4b71941784569da5a41a" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 36143 } } peers { permanent_uuid: "99144cb9a1414f9b80ae8a70887961cb" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 44863 } }
I20260925 04:48:16.544322 17764 heartbeater.cc:503] Master 127.0.98.126:46349 was elected leader, sending a full tablet report...
I20260925 04:48:16.544631 17916 ts_tablet_manager.cc:1428] T e759703c32a344b08da7bd8a966365ea P d822421a171a4b71941784569da5a41a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 04:48:16.545648 17917 ts_tablet_manager.cc:1428] T e759703c32a344b08da7bd8a966365ea P 4183ce246ebd4981ab49d5a023bc9bef: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260925 04:48:16.545955 17898 heartbeater.cc:503] Master 127.0.98.126:46349 was elected leader, sending a full tablet report...
I20260925 04:48:16.545970 17922 raft_consensus.cc:482] T e759703c32a344b08da7bd8a966365ea P d822421a171a4b71941784569da5a41a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:48:16.546209 17922 raft_consensus.cc:504] T e759703c32a344b08da7bd8a966365ea P d822421a171a4b71941784569da5a41a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4183ce246ebd4981ab49d5a023bc9bef" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43279 } } peers { permanent_uuid: "d822421a171a4b71941784569da5a41a" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 36143 } } peers { permanent_uuid: "99144cb9a1414f9b80ae8a70887961cb" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 44863 } }
I20260925 04:48:16.547557 17922 leader_election.cc:288] T e759703c32a344b08da7bd8a966365ea P d822421a171a4b71941784569da5a41a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4183ce246ebd4981ab49d5a023bc9bef (127.0.98.67:43279), 99144cb9a1414f9b80ae8a70887961cb (127.0.98.65:44863)
I20260925 04:48:16.552069 17766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:16.547734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:43279 (local address 127.0.98.66:38691)
0925 04:48:16.547995 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:16.548011 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:48:16.548096 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:48:16.548594 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:48:16.548596 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:48:16.548613 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:48:16.548754 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:16.548761 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:16.549692 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:16.549695 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:16.550972 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:16.550983 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:16.551679 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:16.551683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:16.551803 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:16.551836 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 04:48:16.551904 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 04:48:16.551965 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178}
I20260925 04:48:16.552156 17911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:16.548246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:38691 (local address 127.0.98.67:43279)
0925 04:48:16.548378 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:16.548384 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:16.548399 (+ 15us) server_negotiation.cc:407] Connection header received
0925 04:48:16.548448 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:16.548452 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:16.548499 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:16.548581 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:48:16.548897 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:16.549590 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:16.551116 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:16.551552 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:16.551644 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:16.551695 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:16.551963 (+ 268us) server_negotiation.cc:299] Negotiation successful
0925 04:48:16.552038 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 04:48:16.552810 17852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e759703c32a344b08da7bd8a966365ea" candidate_uuid: "d822421a171a4b71941784569da5a41a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4183ce246ebd4981ab49d5a023bc9bef" is_pre_election: true
I20260925 04:48:16.552973 17924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:16.548512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:44863 (local address 127.0.98.66:41073)
0925 04:48:16.548864 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:16.548879 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:48:16.548954 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:48:16.549348 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:48:16.549351 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:48:16.549368 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:48:16.549480 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:16.549486 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:16.550486 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:16.550489 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:16.551569 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:16.551579 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:16.552521 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:16.552525 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:16.552623 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:16.552643 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 04:48:16.552741 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 04:48:16.552849 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":3,"thread_start_us":120,"threads_started":1}
I20260925 04:48:16.552958 17914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:16.548980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:41073 (local address 127.0.98.65:44863)
0925 04:48:16.549106 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:16.549111 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:16.549124 (+ 13us) server_negotiation.cc:407] Connection header received
0925 04:48:16.549170 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:16.549173 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:16.549222 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:16.549325 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:48:16.549585 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:16.550359 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:16.551776 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:16.552376 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:16.552504 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:16.552564 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:16.552756 (+ 192us) server_negotiation.cc:299] Negotiation successful
0925 04:48:16.552836 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 04:48:16.553154 17852 raft_consensus.cc:2454] T e759703c32a344b08da7bd8a966365ea P 4183ce246ebd4981ab49d5a023bc9bef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d822421a171a4b71941784569da5a41a in term 0.
I20260925 04:48:16.553747 17651 leader_election.cc:302] T e759703c32a344b08da7bd8a966365ea P d822421a171a4b71941784569da5a41a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4183ce246ebd4981ab49d5a023bc9bef, d822421a171a4b71941784569da5a41a; no voters:
I20260925 04:48:16.553728 17584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e759703c32a344b08da7bd8a966365ea" candidate_uuid: "d822421a171a4b71941784569da5a41a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99144cb9a1414f9b80ae8a70887961cb" is_pre_election: true
I20260925 04:48:16.554064 17922 raft_consensus.cc:2781] T e759703c32a344b08da7bd8a966365ea P d822421a171a4b71941784569da5a41a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:48:16.554144 17922 raft_consensus.cc:482] T e759703c32a344b08da7bd8a966365ea P d822421a171a4b71941784569da5a41a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:48:16.554075 17584 raft_consensus.cc:2454] T e759703c32a344b08da7bd8a966365ea P 99144cb9a1414f9b80ae8a70887961cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d822421a171a4b71941784569da5a41a in term 0.
I20260925 04:48:16.554176 17922 raft_consensus.cc:3037] T e759703c32a344b08da7bd8a966365ea P d822421a171a4b71941784569da5a41a [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:48:16.555397 17922 raft_consensus.cc:504] T e759703c32a344b08da7bd8a966365ea P d822421a171a4b71941784569da5a41a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4183ce246ebd4981ab49d5a023bc9bef" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43279 } } peers { permanent_uuid: "d822421a171a4b71941784569da5a41a" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 36143 } } peers { permanent_uuid: "99144cb9a1414f9b80ae8a70887961cb" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 44863 } }
I20260925 04:48:16.555787 17922 leader_election.cc:288] T e759703c32a344b08da7bd8a966365ea P d822421a171a4b71941784569da5a41a [CANDIDATE]: Term 1 election: Requested vote from peers 4183ce246ebd4981ab49d5a023bc9bef (127.0.98.67:43279), 99144cb9a1414f9b80ae8a70887961cb (127.0.98.65:44863)
I20260925 04:48:16.556173 17584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e759703c32a344b08da7bd8a966365ea" candidate_uuid: "d822421a171a4b71941784569da5a41a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99144cb9a1414f9b80ae8a70887961cb"
I20260925 04:48:16.556190 17852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e759703c32a344b08da7bd8a966365ea" candidate_uuid: "d822421a171a4b71941784569da5a41a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4183ce246ebd4981ab49d5a023bc9bef"
I20260925 04:48:16.556314 17584 raft_consensus.cc:3037] T e759703c32a344b08da7bd8a966365ea P 99144cb9a1414f9b80ae8a70887961cb [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:48:16.556317 17852 raft_consensus.cc:3037] T e759703c32a344b08da7bd8a966365ea P 4183ce246ebd4981ab49d5a023bc9bef [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:48:16.557458 17852 raft_consensus.cc:2454] T e759703c32a344b08da7bd8a966365ea P 4183ce246ebd4981ab49d5a023bc9bef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d822421a171a4b71941784569da5a41a in term 1.
I20260925 04:48:16.557662 17584 raft_consensus.cc:2454] T e759703c32a344b08da7bd8a966365ea P 99144cb9a1414f9b80ae8a70887961cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d822421a171a4b71941784569da5a41a in term 1.
I20260925 04:48:16.557816 17651 leader_election.cc:302] T e759703c32a344b08da7bd8a966365ea P d822421a171a4b71941784569da5a41a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4183ce246ebd4981ab49d5a023bc9bef, d822421a171a4b71941784569da5a41a; no voters:
I20260925 04:48:16.558022 17922 raft_consensus.cc:2781] T e759703c32a344b08da7bd8a966365ea P d822421a171a4b71941784569da5a41a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:48:16.558300 17922 raft_consensus.cc:686] T e759703c32a344b08da7bd8a966365ea P d822421a171a4b71941784569da5a41a [term 1 LEADER]: Becoming Leader. State: Replica: d822421a171a4b71941784569da5a41a, State: Running, Role: LEADER
I20260925 04:48:16.558646 17922 consensus_queue.cc:237] T e759703c32a344b08da7bd8a966365ea P d822421a171a4b71941784569da5a41a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4183ce246ebd4981ab49d5a023bc9bef" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43279 } } peers { permanent_uuid: "d822421a171a4b71941784569da5a41a" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 36143 } } peers { permanent_uuid: "99144cb9a1414f9b80ae8a70887961cb" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 44863 } }
I20260925 04:48:16.562024 17438 catalog_manager.cc:5495] T e759703c32a344b08da7bd8a966365ea P d822421a171a4b71941784569da5a41a reported cstate change: term changed from 0 to 1, leader changed from <none> to d822421a171a4b71941784569da5a41a (127.0.98.66). New cstate: current_term: 1 leader_uuid: "d822421a171a4b71941784569da5a41a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4183ce246ebd4981ab49d5a023bc9bef" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d822421a171a4b71941784569da5a41a" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 36143 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99144cb9a1414f9b80ae8a70887961cb" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 44863 } health_report { overall_health: UNKNOWN } } }
I20260925 04:48:16.576145 17913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:16.572102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58058 (local address 127.0.98.66:36143)
0925 04:48:16.572253 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:16.572257 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:16.572271 (+ 14us) server_negotiation.cc:407] Connection header received
0925 04:48:16.572423 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:16.572426 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:16.572484 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:16.572580 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:48:16.572840 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:16.573398 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:16.574536 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:16.574777 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:16.574866 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:16.575214 (+ 348us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:48:16.575675 (+ 461us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:48:16.575771 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:16.575901 (+ 130us) server_negotiation.cc:299] Negotiation successful
0925 04:48:16.575960 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 04:48:16.582151 17584 raft_consensus.cc:1261] T e759703c32a344b08da7bd8a966365ea P 99144cb9a1414f9b80ae8a70887961cb [term 1 FOLLOWER]: Refusing update from remote peer d822421a171a4b71941784569da5a41a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:48:16.582151 17852 raft_consensus.cc:1261] T e759703c32a344b08da7bd8a966365ea P 4183ce246ebd4981ab49d5a023bc9bef [term 1 FOLLOWER]: Refusing update from remote peer d822421a171a4b71941784569da5a41a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:48:16.583144 17922 consensus_queue.cc:1035] T e759703c32a344b08da7bd8a966365ea P d822421a171a4b71941784569da5a41a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99144cb9a1414f9b80ae8a70887961cb" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 44863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:48:16.583365 17925 consensus_queue.cc:1035] T e759703c32a344b08da7bd8a966365ea P d822421a171a4b71941784569da5a41a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4183ce246ebd4981ab49d5a023bc9bef" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:48:16.590943 17928 mvcc.cc:205] Tried to move back new op lower bound from 7333116709193003008 to 7333116709107191808. Current Snapshot: MvccSnapshot[applied={T|T < 7333116709193003008}]
I20260925 04:48:16.591219 17930 mvcc.cc:205] Tried to move back new op lower bound from 7333116709193003008 to 7333116709107191808. Current Snapshot: MvccSnapshot[applied={T|T < 7333116709193003008}]
I20260925 04:48:16.601850 17439 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:48:16.602072 17439 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:48:16.604609 17439 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f3b25b4a67c64cc48e125eeb4a3bc1fb: Sending DeleteTablet for 3 replicas of tablet e759703c32a344b08da7bd8a966365ea
I20260925 04:48:16.605454 17698 tablet_service.cc:1514] Processing DeleteTablet for tablet e759703c32a344b08da7bd8a966365ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:48:16 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:58054
I20260925 04:48:16.605464 17832 tablet_service.cc:1514] Processing DeleteTablet for tablet e759703c32a344b08da7bd8a966365ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:48:16 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:33674
I20260925 04:48:16.605482 17564 tablet_service.cc:1514] Processing DeleteTablet for tablet e759703c32a344b08da7bd8a966365ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:48:16 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:41644
I20260925 04:48:16.606333 17939 tablet_replica.cc:330] T e759703c32a344b08da7bd8a966365ea P d822421a171a4b71941784569da5a41a: stopping tablet replica
I20260925 04:48:16.606698 17939 raft_consensus.cc:2229] T e759703c32a344b08da7bd8a966365ea P d822421a171a4b71941784569da5a41a [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:48:16.607039 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 17501
I20260925 04:48:16.607167 17939 raft_consensus.cc:2258] T e759703c32a344b08da7bd8a966365ea P d822421a171a4b71941784569da5a41a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:48:16.607826 17937 tablet_replica.cc:330] T e759703c32a344b08da7bd8a966365ea P 4183ce246ebd4981ab49d5a023bc9bef: stopping tablet replica
I20260925 04:48:16.608150 17937 raft_consensus.cc:2229] T e759703c32a344b08da7bd8a966365ea P 4183ce246ebd4981ab49d5a023bc9bef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:48:16.608556 17937 raft_consensus.cc:2258] T e759703c32a344b08da7bd8a966365ea P 4183ce246ebd4981ab49d5a023bc9bef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:48:16.609870 17939 ts_tablet_manager.cc:1905] T e759703c32a344b08da7bd8a966365ea P d822421a171a4b71941784569da5a41a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:48:16.611188 17937 ts_tablet_manager.cc:1905] T e759703c32a344b08da7bd8a966365ea P 4183ce246ebd4981ab49d5a023bc9bef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:48:16.613667 17939 ts_tablet_manager.cc:1918] T e759703c32a344b08da7bd8a966365ea P d822421a171a4b71941784569da5a41a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:48:16.613806 17937 ts_tablet_manager.cc:1918] T e759703c32a344b08da7bd8a966365ea P 4183ce246ebd4981ab49d5a023bc9bef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:48:16.613905 17937 log.cc:1196] T e759703c32a344b08da7bd8a966365ea P 4183ce246ebd4981ab49d5a023bc9bef: Deleting WAL directory at /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/minicluster-data/ts-2/wal/wals/e759703c32a344b08da7bd8a966365ea
I20260925 04:48:16.613857 17939 log.cc:1196] T e759703c32a344b08da7bd8a966365ea P d822421a171a4b71941784569da5a41a: Deleting WAL directory at /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790311449873301-393-0/minicluster-data/ts-1/wal/wals/e759703c32a344b08da7bd8a966365ea
I20260925 04:48:16.614265 17937 ts_tablet_manager.cc:1939] T e759703c32a344b08da7bd8a966365ea P 4183ce246ebd4981ab49d5a023bc9bef: Deleting consensus metadata
I20260925 04:48:16.614442 17939 ts_tablet_manager.cc:1939] T e759703c32a344b08da7bd8a966365ea P d822421a171a4b71941784569da5a41a: Deleting consensus metadata
I20260925 04:48:16.615566 17423 catalog_manager.cc:4841] TS 4183ce246ebd4981ab49d5a023bc9bef (127.0.98.67:43279): tablet e759703c32a344b08da7bd8a966365ea (table test-table [id=c5c95151cc0c43acb6b5f44fab89abb0]) successfully deleted
I20260925 04:48:16.615631 17426 catalog_manager.cc:4841] TS d822421a171a4b71941784569da5a41a (127.0.98.66:36143): tablet e759703c32a344b08da7bd8a966365ea (table test-table [id=c5c95151cc0c43acb6b5f44fab89abb0]) successfully deleted
W20260925 04:48:16.615810 17425 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.65:44863 (error 108)
I20260925 04:48:16.616725 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 17635
I20260925 04:48:16.616940 17910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:16.616598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:44863 (local address 127.0.0.1:41654)
0925 04:48:16.616753 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:16.616836 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.65:44863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260925 04:48:16.617810 17425 catalog_manager.cc:4584] TS 99144cb9a1414f9b80ae8a70887961cb (127.0.98.65:44863): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e759703c32a344b08da7bd8a966365ea: Network error: Client connection negotiation failed: client connection to 127.0.98.65:44863: connect: Connection refused (error 111)
I20260925 04:48:16.622172 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 17769
I20260925 04:48:16.627382 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 17407
2026-09-25T04:48:16Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2241 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-25T04:48:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:48:16Z Disabled control of system clock
I20260925 04:48:16.657034 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311449873301-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311449873301-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311449873301-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:34013
--webserver_interface=127.0.98.126
--webserver_port=0
--builtin_ntp_servers=127.0.98.84:36011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:34013
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 04:48:16.766848 17944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:48:16.767118 17944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:48:16.767155 17944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:48:16.771116 17944 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:48:16.771175 17944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:48:16.771190 17944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:48:16.771207 17944 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:48:16.771222 17944 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:48:16.776227 17944 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:36011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311449873301-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311449873301-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:34013
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:34013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17944
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311449873301-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:48:16.777272 17944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:48:16.778718 17944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:48:16.787407 17952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:48:16.787449 17944 server_base.cc:1034] running on GCE node
W20260925 04:48:16.787389 17949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:48:16.787380 17950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:48:16.788192 17944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:48:16.789309 17944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:48:16.790469 17944 hybrid_clock.cc:648] HybridClock initialized: now 1790311696790450 us; error 32 us; skew 500 ppm
I20260925 04:48:16.792294 17944 webserver.cc:460] Webserver started at http://127.0.98.126:34283/ using document root <none> and password file <none>
I20260925 04:48:16.792902 17944 fs_manager.cc:359] Metadata directory not provided
I20260925 04:48:16.792965 17944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:48:16.793185 17944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:48:16.795290 17944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311449873301-393-0/minicluster-data/master-0/data/instance:
uuid: "3d8b320ec056484991827ba1b1e13a5f"
format_stamp: "Formatted at 2026-09-25 04:48:16 on dist-test-slave-kghz"
server_key: "f618905757d9d647c6ba0f446551c69c"
server_key_iv: "f28e2bd28172750d5d1dd23b37c492ed"
server_key_version: "encryptionkey@0"
I20260925 04:48:16.795873 17944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311449873301-393-0/minicluster-data/master-0/wal/instance:
uuid: "3d8b320ec056484991827ba1b1e13a5f"
format_stamp: "Formatted at 2026-09-25 04:48:16 on dist-test-slave-kghz"
server_key: "f618905757d9d647c6ba0f446551c69c"
server_key_iv: "f28e2bd28172750d5d1dd23b37c492ed"
server_key_version: "encryptionkey@0"
I20260925 04:48:16.799597 17944 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 04:48:16.802156 17958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:48:16.803640 17944 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 04:48:16.803754 17944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311449873301-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311449873301-393-0/minicluster-data/master-0/wal
uuid: "3d8b320ec056484991827ba1b1e13a5f"
format_stamp: "Formatted at 2026-09-25 04:48:16 on dist-test-slave-kghz"
server_key: "f618905757d9d647c6ba0f446551c69c"
server_key_iv: "f28e2bd28172750d5d1dd23b37c492ed"
server_key_version: "encryptionkey@0"
I20260925 04:48:16.803920 17944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311449873301-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311449873301-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311449873301-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:48:16.828220 17944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:48:16.828996 17944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:48:16.829201 17944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:48:16.836231 17944 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:34013
I20260925 04:48:16.836246 18010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:34013 every 8 connection(s)
I20260925 04:48:16.837183 17944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
I20260925 04:48:16.840286 18011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:48:16.843245 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 17944
I20260925 04:48:16.843374 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311449873301-393-0/minicluster-data/master-0/wal/instance
I20260925 04:48:16.843675 393 external_mini_cluster.cc:1428] Setting key dc32ba7d7df3fc6dec90256e4f7becb6
I20260925 04:48:16.846645 18011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d8b320ec056484991827ba1b1e13a5f: Bootstrap starting.
I20260925 04:48:16.848946 18011 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d8b320ec056484991827ba1b1e13a5f: Neither blocks nor log segments found. Creating new log.
I20260925 04:48:16.849679 18011 log.cc:824] T 00000000000000000000000000000000 P 3d8b320ec056484991827ba1b1e13a5f: Log is configured to *not* fsync() on all Append() calls
I20260925 04:48:16.851649 18011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d8b320ec056484991827ba1b1e13a5f: No bootstrap required, opened a new log
I20260925 04:48:16.853346 18014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:16.845234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57378 (local address 127.0.98.126:34013)
0925 04:48:16.845890 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:16.845910 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:16.845947 (+ 37us) server_negotiation.cc:407] Connection header received
0925 04:48:16.846665 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:16.846684 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:16.847134 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:16.847485 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:16.847943 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:16.848899 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:16.849802 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:16.850488 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:16.850680 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:16.851588 (+ 908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:16.851593 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:16.851605 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:48:16.852054 (+ 449us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:16.852112 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:48:16.852132 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:16.852240 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:16.852376 (+ 136us) server_negotiation.cc:299] Negotiation successful
0925 04:48:16.852645 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":141,"threads_started":1}
I20260925 04:48:16.855686 18011 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d8b320ec056484991827ba1b1e13a5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d8b320ec056484991827ba1b1e13a5f" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34013 } }
I20260925 04:48:16.855933 18011 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d8b320ec056484991827ba1b1e13a5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:48:16.856015 18011 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d8b320ec056484991827ba1b1e13a5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d8b320ec056484991827ba1b1e13a5f, State: Initialized, Role: FOLLOWER
I20260925 04:48:16.856462 18011 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d8b320ec056484991827ba1b1e13a5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d8b320ec056484991827ba1b1e13a5f" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34013 } }
I20260925 04:48:16.856621 18011 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d8b320ec056484991827ba1b1e13a5f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:48:16.856705 18011 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d8b320ec056484991827ba1b1e13a5f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:48:16.856814 18011 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d8b320ec056484991827ba1b1e13a5f [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:48:16.857726 18011 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d8b320ec056484991827ba1b1e13a5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d8b320ec056484991827ba1b1e13a5f" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34013 } }
I20260925 04:48:16.858065 18011 leader_election.cc:302] T 00000000000000000000000000000000 P 3d8b320ec056484991827ba1b1e13a5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d8b320ec056484991827ba1b1e13a5f; no voters:
I20260925 04:48:16.858453 18011 leader_election.cc:288] T 00000000000000000000000000000000 P 3d8b320ec056484991827ba1b1e13a5f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:48:16.858526 18016 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d8b320ec056484991827ba1b1e13a5f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:48:16.858783 18016 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d8b320ec056484991827ba1b1e13a5f [term 1 LEADER]: Becoming Leader. State: Replica: 3d8b320ec056484991827ba1b1e13a5f, State: Running, Role: LEADER
I20260925 04:48:16.859083 18016 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d8b320ec056484991827ba1b1e13a5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d8b320ec056484991827ba1b1e13a5f" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34013 } }
I20260925 04:48:16.859683 18011 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d8b320ec056484991827ba1b1e13a5f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:48:16.860790 18017 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d8b320ec056484991827ba1b1e13a5f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d8b320ec056484991827ba1b1e13a5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d8b320ec056484991827ba1b1e13a5f" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34013 } } }
I20260925 04:48:16.860935 18017 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d8b320ec056484991827ba1b1e13a5f [sys.catalog]: This master's current role is: LEADER
I20260925 04:48:16.861296 18025 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:48:16.862401 18018 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d8b320ec056484991827ba1b1e13a5f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d8b320ec056484991827ba1b1e13a5f. Latest consensus state: current_term: 1 leader_uuid: "3d8b320ec056484991827ba1b1e13a5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d8b320ec056484991827ba1b1e13a5f" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34013 } } }
I20260925 04:48:16.862495 18018 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d8b320ec056484991827ba1b1e13a5f [sys.catalog]: This master's current role is: LEADER
I20260925 04:48:16.864166 18025 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:48:16.869763 18025 catalog_manager.cc:1347] Generated new cluster ID: 14cb5ab6ddce4c3286d7157c4befb7fa
I20260925 04:48:16.869848 18025 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:48:16.892333 18025 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:48:16.893287 18025 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:48:16.898523 18025 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d8b320ec056484991827ba1b1e13a5f: Generated new TSK 0
I20260925 04:48:16.899212 18025 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:48:16.907372 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311449873301-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311449873301-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311449873301-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:0
--local_ip_for_outbound_sockets=127.0.98.65
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:34013
--builtin_ntp_servers=127.0.98.84:36011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 04:48:17.031303 18035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:48:17.031571 18035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:48:17.031611 18035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:48:17.035734 18035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:48:17.035785 18035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:48:17.035871 18035 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260925 04:48:17.040608 18035 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:36011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311449873301-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311449873301-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:34013
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18035
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311449873301-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:48:17.041692 18035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:48:17.043107 18035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:48:17.052814 18043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:48:17.052819 18040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:48:17.052822 18041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:48:17.053212 18035 server_base.cc:1034] running on GCE node
I20260925 04:48:17.053673 18035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:48:17.054363 18035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:48:17.055533 18035 hybrid_clock.cc:648] HybridClock initialized: now 1790311697055512 us; error 35 us; skew 500 ppm
I20260925 04:48:17.057304 18035 webserver.cc:460] Webserver started at http://127.0.98.65:37807/ using document root <none> and password file <none>
I20260925 04:48:17.057896 18035 fs_manager.cc:359] Metadata directory not provided
I20260925 04:48:17.057960 18035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:48:17.058174 18035 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:48:17.060308 18035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311449873301-393-0/minicluster-data/ts-0/data/instance:
uuid: "cb855c3e9c83417aa24d7aee6043edd1"
format_stamp: "Formatted at 2026-09-25 04:48:17 on dist-test-slave-kghz"
server_key: "becbeb7037336743ecf50f77495e9b5d"
server_key_iv: "23d98f4bd613a5ecb64fcfca2e3149e1"
server_key_version: "encryptionkey@0"
I20260925 04:48:17.060910 18035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311449873301-393-0/minicluster-data/ts-0/wal/instance:
uuid: "cb855c3e9c83417aa24d7aee6043edd1"
format_stamp: "Formatted at 2026-09-25 04:48:17 on dist-test-slave-kghz"
server_key: "becbeb7037336743ecf50f77495e9b5d"
server_key_iv: "23d98f4bd613a5ecb64fcfca2e3149e1"
server_key_version: "encryptionkey@0"
I20260925 04:48:17.064716 18035 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 04:48:17.067113 18049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:48:17.068287 18035 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 04:48:17.068434 18035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311449873301-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311449873301-393-0/minicluster-data/ts-0/wal
uuid: "cb855c3e9c83417aa24d7aee6043edd1"
format_stamp: "Formatted at 2026-09-25 04:48:17 on dist-test-slave-kghz"
server_key: "becbeb7037336743ecf50f77495e9b5d"
server_key_iv: "23d98f4bd613a5ecb64fcfca2e3149e1"
server_key_version: "encryptionkey@0"
I20260925 04:48:17.068547 18035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311449873301-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311449873301-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311449873301-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:48:17.083421 18035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:48:17.084148 18035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:48:17.084352 18035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:48:17.084924 18035 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:48:17.085867 18035 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:48:17.085942 18035 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:48:17.086009 18035 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:48:17.086053 18035 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:48:17.094842 18035 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:41973
I20260925 04:48:17.094868 18162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:41973 every 8 connection(s)
I20260925 04:48:17.095767 18035 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
I20260925 04:48:17.104465 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 18035
I20260925 04:48:17.104595 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311449873301-393-0/minicluster-data/ts-0/wal/instance
I20260925 04:48:17.104975 393 external_mini_cluster.cc:1428] Setting key 94e1c15a1d194d69c6df255d6374b177
I20260925 04:48:17.105400 18014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:17.097728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:49487 (local address 127.0.98.126:34013)
0925 04:48:17.097875 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:17.097879 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:17.098974 (+ 1095us) server_negotiation.cc:407] Connection header received
0925 04:48:17.100419 (+ 1445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:17.100422 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:17.100483 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:17.100584 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:17.101503 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:17.102054 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:17.103273 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:17.103515 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:17.103625 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:17.104825 (+ 1200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:17.104828 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:17.104830 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:48:17.104849 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:17.104869 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:48:17.104878 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:17.104938 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:17.105192 (+ 254us) server_negotiation.cc:299] Negotiation successful
0925 04:48:17.105246 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 04:48:17.106527 18165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:17.098133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34013 (local address 127.0.98.65:49487)
0925 04:48:17.098810 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:17.098851 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 04:48:17.100177 (+ 1326us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:48:17.100762 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:48:17.100769 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:48:17.100855 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:48:17.101356 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:17.101370 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:17.102193 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:17.102197 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:17.103141 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:17.103151 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:17.103665 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:17.103669 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:17.103832 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:17.104498 (+ 666us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:48:17.104500 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:48:17.104560 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:48:17.104647 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:48:17.104966 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:48:17.104975 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:48:17.104982 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 04:48:17.105162 (+ 180us) client_negotiation.cc:240] Negotiation successful
0925 04:48:17.105486 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":133,"threads_started":1}
I20260925 04:48:17.107236 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311449873301-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311449873301-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311449873301-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:0
--local_ip_for_outbound_sockets=127.0.98.66
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:34013
--builtin_ntp_servers=127.0.98.84:36011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 04:48:17.107867 18163 heartbeater.cc:347] Connected to a master server at 127.0.98.126:34013
I20260925 04:48:17.108107 18163 heartbeater.cc:464] Registering TS with master...
I20260925 04:48:17.108800 18163 heartbeater.cc:511] Master 127.0.98.126:34013 requested a full tablet report, sending...
I20260925 04:48:17.110461 17975 ts_manager.cc:194] Registered new tserver with Master: cb855c3e9c83417aa24d7aee6043edd1 (127.0.98.65:41973)
I20260925 04:48:17.112195 17975 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.65:49487
W20260925 04:48:17.221414 18166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:48:17.221688 18166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:48:17.221727 18166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:48:17.225761 18166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:48:17.225832 18166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:48:17.225889 18166 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260925 04:48:17.230613 18166 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:36011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311449873301-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311449873301-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:34013
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18166
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311449873301-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:48:17.231678 18166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:48:17.233016 18166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:48:17.242568 18171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:48:17.242568 18172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:48:17.242759 18166 server_base.cc:1034] running on GCE node
W20260925 04:48:17.242626 18174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:48:17.243263 18166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:48:17.243863 18166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:48:17.245043 18166 hybrid_clock.cc:648] HybridClock initialized: now 1790311697245019 us; error 34 us; skew 500 ppm
I20260925 04:48:17.247223 18166 webserver.cc:460] Webserver started at http://127.0.98.66:44197/ using document root <none> and password file <none>
I20260925 04:48:17.247817 18166 fs_manager.cc:359] Metadata directory not provided
I20260925 04:48:17.247880 18166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:48:17.248096 18166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:48:17.250174 18166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311449873301-393-0/minicluster-data/ts-1/data/instance:
uuid: "83a9e539db4444f697f95f180c02ddb9"
format_stamp: "Formatted at 2026-09-25 04:48:17 on dist-test-slave-kghz"
server_key: "7a3db4ce7f1b63f20c55f1698513a9b4"
server_key_iv: "1e6e803d3b0e91c6349b2768829fa94f"
server_key_version: "encryptionkey@0"
I20260925 04:48:17.250814 18166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311449873301-393-0/minicluster-data/ts-1/wal/instance:
uuid: "83a9e539db4444f697f95f180c02ddb9"
format_stamp: "Formatted at 2026-09-25 04:48:17 on dist-test-slave-kghz"
server_key: "7a3db4ce7f1b63f20c55f1698513a9b4"
server_key_iv: "1e6e803d3b0e91c6349b2768829fa94f"
server_key_version: "encryptionkey@0"
I20260925 04:48:17.254616 18166 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 04:48:17.257045 18180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:48:17.258368 18166 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 04:48:17.258497 18166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311449873301-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311449873301-393-0/minicluster-data/ts-1/wal
uuid: "83a9e539db4444f697f95f180c02ddb9"
format_stamp: "Formatted at 2026-09-25 04:48:17 on dist-test-slave-kghz"
server_key: "7a3db4ce7f1b63f20c55f1698513a9b4"
server_key_iv: "1e6e803d3b0e91c6349b2768829fa94f"
server_key_version: "encryptionkey@0"
I20260925 04:48:17.258610 18166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311449873301-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311449873301-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311449873301-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:48:17.276258 18166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:48:17.276957 18166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:48:17.277160 18166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:48:17.277786 18166 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:48:17.278801 18166 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:48:17.278848 18166 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:48:17.278920 18166 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:48:17.278956 18166 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:48:17.288146 18166 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:34313
I20260925 04:48:17.288165 18293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:34313 every 8 connection(s)
I20260925 04:48:17.289067 18166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
I20260925 04:48:17.294489 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 18166
I20260925 04:48:17.294625 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311449873301-393-0/minicluster-data/ts-1/wal/instance
I20260925 04:48:17.294945 393 external_mini_cluster.cc:1428] Setting key 50179ee4553149d8267fdb43af39839e
I20260925 04:48:17.297113 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311449873301-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311449873301-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311449873301-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:0
--local_ip_for_outbound_sockets=127.0.98.67
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:34013
--builtin_ntp_servers=127.0.98.84:36011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 04:48:17.299716 18014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:17.291005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:40715 (local address 127.0.98.126:34013)
0925 04:48:17.291153 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:17.291157 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:17.292308 (+ 1151us) server_negotiation.cc:407] Connection header received
0925 04:48:17.293773 (+ 1465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:17.293776 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:17.293837 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:17.293952 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:17.295190 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:17.295878 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:17.297442 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:17.297667 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:17.297754 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:17.299217 (+ 1463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:17.299220 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:17.299222 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:48:17.299234 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:17.299251 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:48:17.299259 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:17.299339 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:17.299519 (+ 180us) server_negotiation.cc:299] Negotiation successful
0925 04:48:17.299578 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 04:48:17.300577 18296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:17.291400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34013 (local address 127.0.98.66:40715)
0925 04:48:17.292145 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:17.292189 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 04:48:17.293530 (+ 1341us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:48:17.294125 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:48:17.294132 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:48:17.294298 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:48:17.295031 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:17.295052 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:17.296011 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:17.296014 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:17.297261 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:17.297277 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:17.297834 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:17.297841 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:17.298094 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:17.298917 (+ 823us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:48:17.298919 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:48:17.298960 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:48:17.299024 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:48:17.299354 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:48:17.299364 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:48:17.299376 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 04:48:17.299508 (+ 132us) client_negotiation.cc:240] Negotiation successful
0925 04:48:17.299809 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":109,"threads_started":1}
I20260925 04:48:17.301877 18294 heartbeater.cc:347] Connected to a master server at 127.0.98.126:34013
I20260925 04:48:17.302110 18294 heartbeater.cc:464] Registering TS with master...
I20260925 04:48:17.302744 18294 heartbeater.cc:511] Master 127.0.98.126:34013 requested a full tablet report, sending...
I20260925 04:48:17.303818 17975 ts_manager.cc:194] Registered new tserver with Master: 83a9e539db4444f697f95f180c02ddb9 (127.0.98.66:34313)
I20260925 04:48:17.304376 17975 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.66:40715
W20260925 04:48:17.409973 18297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:48:17.410295 18297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:48:17.410370 18297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:48:17.414462 18297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:48:17.414512 18297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:48:17.414595 18297 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260925 04:48:17.419432 18297 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:36011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311449873301-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311449873301-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:34013
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18297
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311449873301-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:48:17.420539 18297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:48:17.421960 18297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:48:17.431669 18302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:48:17.431680 18305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:48:17.431797 18297 server_base.cc:1034] running on GCE node
W20260925 04:48:17.431686 18303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:48:17.432349 18297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:48:17.433063 18297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:48:17.434337 18297 hybrid_clock.cc:648] HybridClock initialized: now 1790311697434319 us; error 28 us; skew 500 ppm
I20260925 04:48:17.436254 18297 webserver.cc:460] Webserver started at http://127.0.98.67:41647/ using document root <none> and password file <none>
I20260925 04:48:17.436908 18297 fs_manager.cc:359] Metadata directory not provided
I20260925 04:48:17.437006 18297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:48:17.437242 18297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:48:17.439587 18297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311449873301-393-0/minicluster-data/ts-2/data/instance:
uuid: "28f59bbd6ffb4986b78211be1bcd85f6"
format_stamp: "Formatted at 2026-09-25 04:48:17 on dist-test-slave-kghz"
server_key: "ec5c42e885f1991569184a6c26800ec8"
server_key_iv: "ea8e87593e2bd01eb325a99a18dedabe"
server_key_version: "encryptionkey@0"
I20260925 04:48:17.440179 18297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311449873301-393-0/minicluster-data/ts-2/wal/instance:
uuid: "28f59bbd6ffb4986b78211be1bcd85f6"
format_stamp: "Formatted at 2026-09-25 04:48:17 on dist-test-slave-kghz"
server_key: "ec5c42e885f1991569184a6c26800ec8"
server_key_iv: "ea8e87593e2bd01eb325a99a18dedabe"
server_key_version: "encryptionkey@0"
I20260925 04:48:17.444104 18297 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 04:48:17.446708 18311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:48:17.448091 18297 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:48:17.448231 18297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311449873301-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311449873301-393-0/minicluster-data/ts-2/wal
uuid: "28f59bbd6ffb4986b78211be1bcd85f6"
format_stamp: "Formatted at 2026-09-25 04:48:17 on dist-test-slave-kghz"
server_key: "ec5c42e885f1991569184a6c26800ec8"
server_key_iv: "ea8e87593e2bd01eb325a99a18dedabe"
server_key_version: "encryptionkey@0"
I20260925 04:48:17.448352 18297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311449873301-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311449873301-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311449873301-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:48:17.468928 18297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:48:17.469646 18297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:48:17.469857 18297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:48:17.470532 18297 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:48:17.471467 18297 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:48:17.471537 18297 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:48:17.471611 18297 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:48:17.471662 18297 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:48:17.481021 18297 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:35699
I20260925 04:48:17.481070 18424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:35699 every 8 connection(s)
I20260925 04:48:17.481940 18297 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
I20260925 04:48:17.484040 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 18297
I20260925 04:48:17.484175 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311449873301-393-0/minicluster-data/ts-2/wal/instance
I20260925 04:48:17.484444 393 external_mini_cluster.cc:1428] Setting key c67668c2afdbb33f433260460caa24e2
I20260925 04:48:17.491009 18014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:17.483866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:34429 (local address 127.0.98.126:34013)
0925 04:48:17.484034 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:17.484041 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:17.485146 (+ 1105us) server_negotiation.cc:407] Connection header received
0925 04:48:17.486605 (+ 1459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:17.486608 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:17.486651 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:17.486724 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:17.487519 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:17.488026 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:17.489109 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:17.489337 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:17.489416 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:17.490607 (+ 1191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:17.490610 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:17.490612 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:48:17.490623 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:17.490643 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:48:17.490650 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:17.490719 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:17.490853 (+ 134us) server_negotiation.cc:299] Negotiation successful
0925 04:48:17.490903 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 04:48:17.491820 18427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:17.484296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34013 (local address 127.0.98.67:34429)
0925 04:48:17.484995 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:17.485044 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 04:48:17.486413 (+ 1369us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:48:17.486849 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:48:17.486855 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:48:17.486935 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:48:17.487411 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:17.487423 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:17.488138 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:17.488140 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:17.489001 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:17.489008 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:17.489467 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:17.489469 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:17.489641 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:17.490355 (+ 714us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:48:17.490358 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:48:17.490406 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:48:17.490468 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:48:17.490734 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:48:17.490740 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:48:17.490744 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 04:48:17.490837 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 04:48:17.491041 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":126,"threads_started":1}
I20260925 04:48:17.492834 18425 heartbeater.cc:347] Connected to a master server at 127.0.98.126:34013
I20260925 04:48:17.493057 18425 heartbeater.cc:464] Registering TS with master...
I20260925 04:48:17.493541 18425 heartbeater.cc:511] Master 127.0.98.126:34013 requested a full tablet report, sending...
I20260925 04:48:17.494567 17975 ts_manager.cc:194] Registered new tserver with Master: 28f59bbd6ffb4986b78211be1bcd85f6 (127.0.98.67:35699)
I20260925 04:48:17.495116 17975 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.67:34429
I20260925 04:48:17.498073 393 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 04:48:17.503690 18014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:17.499948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57382 (local address 127.0.98.126:34013)
0925 04:48:17.500089 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:17.500094 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:17.500231 (+ 137us) server_negotiation.cc:407] Connection header received
0925 04:48:17.500387 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:17.500391 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:17.500439 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:17.500522 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:17.500822 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:17.501317 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:17.502174 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:17.502418 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:17.502527 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:17.503294 (+ 767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:17.503298 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:17.503306 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:48:17.503318 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:17.503333 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:48:17.503341 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:17.503419 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:17.503508 (+ 89us) server_negotiation.cc:299] Negotiation successful
0925 04:48:17.503556 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 04:48:17.506307 17975 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57382:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:48:17.509521 17975 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:48:17.523798 18437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:17.517889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:35699 (local address 127.0.0.1:56208)
0925 04:48:17.518693 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:17.518749 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 04:48:17.518890 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:48:17.519492 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:48:17.519498 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:48:17.519521 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:48:17.519648 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:17.519655 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:17.521131 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:17.521134 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:17.522224 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:17.522234 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:17.523218 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:17.523222 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:17.523330 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:17.523372 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 04:48:17.523477 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 04:48:17.523603 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":673,"mutex_wait_us":89,"spinlock_wait_cycles":8064,"thread_start_us":105,"threads_started":1}
I20260925 04:48:17.524216 18439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:17.518536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:41973 (local address 127.0.0.1:45996)
0925 04:48:17.519153 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:17.519181 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0925 04:48:17.519269 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:48:17.519918 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:48:17.519924 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:48:17.519945 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:48:17.520025 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:17.520030 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:17.521308 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:17.521315 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:17.522693 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:17.522703 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:17.523780 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:17.523783 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:17.523877 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:17.523909 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 04:48:17.523997 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 04:48:17.524075 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"mutex_wait_us":61,"thread_start_us":82,"threads_started":1}
I20260925 04:48:17.524298 18441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:17.518949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45996 (local address 127.0.98.65:41973)
0925 04:48:17.519388 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:17.519401 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:17.519450 (+ 49us) server_negotiation.cc:407] Connection header received
0925 04:48:17.519519 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:17.519524 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:17.519739 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:17.519883 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:48:17.520140 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:17.521204 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:17.522824 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:17.523661 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:17.523768 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:17.523938 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:17.524024 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 04:48:17.524133 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":116,"threads_started":1}
I20260925 04:48:17.524397 18438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:17.518090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56208 (local address 127.0.98.67:35699)
0925 04:48:17.518723 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:17.518736 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:17.519038 (+ 302us) server_negotiation.cc:407] Connection header received
0925 04:48:17.519117 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:17.519122 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:17.519329 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:17.519487 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:48:17.519793 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:17.521010 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:17.522416 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:17.523102 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:17.523211 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:17.523401 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:17.524119 (+ 718us) server_negotiation.cc:299] Negotiation successful
0925 04:48:17.524218 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":199,"threads_started":1}
I20260925 04:48:17.525419 18436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:17.517795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:34313 (local address 127.0.0.1:54764)
0925 04:48:17.518694 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:17.518749 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 04:48:17.518890 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:48:17.519754 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:48:17.519758 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:48:17.519776 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:48:17.519871 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:17.519880 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:17.521345 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:17.521347 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:17.522695 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:17.522703 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:17.524999 (+ 2296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:17.525003 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:17.525110 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:17.525135 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 04:48:17.525190 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 04:48:17.525268 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":711,"mutex_wait_us":102,"spinlock_wait_cycles":3840,"thread_start_us":122,"threads_started":1}
I20260925 04:48:17.526369 18440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:17.518877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54764 (local address 127.0.98.66:34313)
0925 04:48:17.519299 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:17.519310 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:17.519333 (+ 23us) server_negotiation.cc:407] Connection header received
0925 04:48:17.519423 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:17.519430 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:17.519613 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:17.519777 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:48:17.520011 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:17.521214 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:17.523482 (+ 2268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:17.524847 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:17.525838 (+ 991us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:17.526039 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:17.526097 (+ 58us) server_negotiation.cc:299] Negotiation successful
0925 04:48:17.526192 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":115,"threads_started":1}
I20260925 04:48:17.527401 18097 tablet_service.cc:1467] Processing CreateTablet for tablet d4c69dcb4423403bb96e9650ab867185 (DEFAULT_TABLE table=test-table [id=6e9354cde23a4c7b872936cd06aa01eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:48:17.527398 18359 tablet_service.cc:1467] Processing CreateTablet for tablet d4c69dcb4423403bb96e9650ab867185 (DEFAULT_TABLE table=test-table [id=6e9354cde23a4c7b872936cd06aa01eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:48:17.528527 18097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4c69dcb4423403bb96e9650ab867185. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:48:17.528527 18359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4c69dcb4423403bb96e9650ab867185. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:48:17.529590 18228 tablet_service.cc:1467] Processing CreateTablet for tablet d4c69dcb4423403bb96e9650ab867185 (DEFAULT_TABLE table=test-table [id=6e9354cde23a4c7b872936cd06aa01eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:48:17.530512 18228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4c69dcb4423403bb96e9650ab867185. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:48:17.535835 18442 tablet_bootstrap.cc:492] T d4c69dcb4423403bb96e9650ab867185 P 83a9e539db4444f697f95f180c02ddb9: Bootstrap starting.
I20260925 04:48:17.537262 18444 tablet_bootstrap.cc:492] T d4c69dcb4423403bb96e9650ab867185 P cb855c3e9c83417aa24d7aee6043edd1: Bootstrap starting.
I20260925 04:48:17.538528 18442 tablet_bootstrap.cc:654] T d4c69dcb4423403bb96e9650ab867185 P 83a9e539db4444f697f95f180c02ddb9: Neither blocks nor log segments found. Creating new log.
I20260925 04:48:17.538555 18443 tablet_bootstrap.cc:492] T d4c69dcb4423403bb96e9650ab867185 P 28f59bbd6ffb4986b78211be1bcd85f6: Bootstrap starting.
I20260925 04:48:17.539388 18444 tablet_bootstrap.cc:654] T d4c69dcb4423403bb96e9650ab867185 P cb855c3e9c83417aa24d7aee6043edd1: Neither blocks nor log segments found. Creating new log.
I20260925 04:48:17.539419 18442 log.cc:824] T d4c69dcb4423403bb96e9650ab867185 P 83a9e539db4444f697f95f180c02ddb9: Log is configured to *not* fsync() on all Append() calls
I20260925 04:48:17.540310 18444 log.cc:824] T d4c69dcb4423403bb96e9650ab867185 P cb855c3e9c83417aa24d7aee6043edd1: Log is configured to *not* fsync() on all Append() calls
I20260925 04:48:17.540467 18443 tablet_bootstrap.cc:654] T d4c69dcb4423403bb96e9650ab867185 P 28f59bbd6ffb4986b78211be1bcd85f6: Neither blocks nor log segments found. Creating new log.
I20260925 04:48:17.541373 18443 log.cc:824] T d4c69dcb4423403bb96e9650ab867185 P 28f59bbd6ffb4986b78211be1bcd85f6: Log is configured to *not* fsync() on all Append() calls
I20260925 04:48:17.541669 18442 tablet_bootstrap.cc:492] T d4c69dcb4423403bb96e9650ab867185 P 83a9e539db4444f697f95f180c02ddb9: No bootstrap required, opened a new log
I20260925 04:48:17.541872 18442 ts_tablet_manager.cc:1397] T d4c69dcb4423403bb96e9650ab867185 P 83a9e539db4444f697f95f180c02ddb9: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 04:48:17.542312 18444 tablet_bootstrap.cc:492] T d4c69dcb4423403bb96e9650ab867185 P cb855c3e9c83417aa24d7aee6043edd1: No bootstrap required, opened a new log
I20260925 04:48:17.542513 18444 ts_tablet_manager.cc:1397] T d4c69dcb4423403bb96e9650ab867185 P cb855c3e9c83417aa24d7aee6043edd1: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260925 04:48:17.543183 18443 tablet_bootstrap.cc:492] T d4c69dcb4423403bb96e9650ab867185 P 28f59bbd6ffb4986b78211be1bcd85f6: No bootstrap required, opened a new log
I20260925 04:48:17.543383 18443 ts_tablet_manager.cc:1397] T d4c69dcb4423403bb96e9650ab867185 P 28f59bbd6ffb4986b78211be1bcd85f6: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 04:48:17.546329 18442 raft_consensus.cc:348] T d4c69dcb4423403bb96e9650ab867185 P 83a9e539db4444f697f95f180c02ddb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83a9e539db4444f697f95f180c02ddb9" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 34313 } } peers { permanent_uuid: "28f59bbd6ffb4986b78211be1bcd85f6" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 35699 } } peers { permanent_uuid: "cb855c3e9c83417aa24d7aee6043edd1" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41973 } }
I20260925 04:48:17.546613 18442 raft_consensus.cc:374] T d4c69dcb4423403bb96e9650ab867185 P 83a9e539db4444f697f95f180c02ddb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:48:17.546666 18442 raft_consensus.cc:729] T d4c69dcb4423403bb96e9650ab867185 P 83a9e539db4444f697f95f180c02ddb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83a9e539db4444f697f95f180c02ddb9, State: Initialized, Role: FOLLOWER
I20260925 04:48:17.546561 18444 raft_consensus.cc:348] T d4c69dcb4423403bb96e9650ab867185 P cb855c3e9c83417aa24d7aee6043edd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83a9e539db4444f697f95f180c02ddb9" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 34313 } } peers { permanent_uuid: "28f59bbd6ffb4986b78211be1bcd85f6" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 35699 } } peers { permanent_uuid: "cb855c3e9c83417aa24d7aee6043edd1" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41973 } }
I20260925 04:48:17.546790 18444 raft_consensus.cc:374] T d4c69dcb4423403bb96e9650ab867185 P cb855c3e9c83417aa24d7aee6043edd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:48:17.546844 18444 raft_consensus.cc:729] T d4c69dcb4423403bb96e9650ab867185 P cb855c3e9c83417aa24d7aee6043edd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb855c3e9c83417aa24d7aee6043edd1, State: Initialized, Role: FOLLOWER
I20260925 04:48:17.546751 18443 raft_consensus.cc:348] T d4c69dcb4423403bb96e9650ab867185 P 28f59bbd6ffb4986b78211be1bcd85f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83a9e539db4444f697f95f180c02ddb9" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 34313 } } peers { permanent_uuid: "28f59bbd6ffb4986b78211be1bcd85f6" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 35699 } } peers { permanent_uuid: "cb855c3e9c83417aa24d7aee6043edd1" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41973 } }
I20260925 04:48:17.547001 18443 raft_consensus.cc:374] T d4c69dcb4423403bb96e9650ab867185 P 28f59bbd6ffb4986b78211be1bcd85f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:48:17.547070 18443 raft_consensus.cc:729] T d4c69dcb4423403bb96e9650ab867185 P 28f59bbd6ffb4986b78211be1bcd85f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28f59bbd6ffb4986b78211be1bcd85f6, State: Initialized, Role: FOLLOWER
I20260925 04:48:17.547158 18442 consensus_queue.cc:260] T d4c69dcb4423403bb96e9650ab867185 P 83a9e539db4444f697f95f180c02ddb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83a9e539db4444f697f95f180c02ddb9" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 34313 } } peers { permanent_uuid: "28f59bbd6ffb4986b78211be1bcd85f6" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 35699 } } peers { permanent_uuid: "cb855c3e9c83417aa24d7aee6043edd1" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41973 } }
I20260925 04:48:17.547263 18444 consensus_queue.cc:260] T d4c69dcb4423403bb96e9650ab867185 P cb855c3e9c83417aa24d7aee6043edd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83a9e539db4444f697f95f180c02ddb9" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 34313 } } peers { permanent_uuid: "28f59bbd6ffb4986b78211be1bcd85f6" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 35699 } } peers { permanent_uuid: "cb855c3e9c83417aa24d7aee6043edd1" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41973 } }
I20260925 04:48:17.547484 18443 consensus_queue.cc:260] T d4c69dcb4423403bb96e9650ab867185 P 28f59bbd6ffb4986b78211be1bcd85f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83a9e539db4444f697f95f180c02ddb9" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 34313 } } peers { permanent_uuid: "28f59bbd6ffb4986b78211be1bcd85f6" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 35699 } } peers { permanent_uuid: "cb855c3e9c83417aa24d7aee6043edd1" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41973 } }
I20260925 04:48:17.547897 18294 heartbeater.cc:503] Master 127.0.98.126:34013 was elected leader, sending a full tablet report...
I20260925 04:48:17.548174 18444 ts_tablet_manager.cc:1428] T d4c69dcb4423403bb96e9650ab867185 P cb855c3e9c83417aa24d7aee6043edd1: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260925 04:48:17.548166 18442 ts_tablet_manager.cc:1428] T d4c69dcb4423403bb96e9650ab867185 P 83a9e539db4444f697f95f180c02ddb9: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260925 04:48:17.548482 18163 heartbeater.cc:503] Master 127.0.98.126:34013 was elected leader, sending a full tablet report...
I20260925 04:48:17.549057 18425 heartbeater.cc:503] Master 127.0.98.126:34013 was elected leader, sending a full tablet report...
I20260925 04:48:17.549540 18443 ts_tablet_manager.cc:1428] T d4c69dcb4423403bb96e9650ab867185 P 28f59bbd6ffb4986b78211be1bcd85f6: Time spent starting tablet: real 0.006s user 0.001s sys 0.005s
I20260925 04:48:17.588502 18448 raft_consensus.cc:482] T d4c69dcb4423403bb96e9650ab867185 P 83a9e539db4444f697f95f180c02ddb9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:48:17.588792 18448 raft_consensus.cc:504] T d4c69dcb4423403bb96e9650ab867185 P 83a9e539db4444f697f95f180c02ddb9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83a9e539db4444f697f95f180c02ddb9" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 34313 } } peers { permanent_uuid: "28f59bbd6ffb4986b78211be1bcd85f6" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 35699 } } peers { permanent_uuid: "cb855c3e9c83417aa24d7aee6043edd1" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41973 } }
I20260925 04:48:17.590015 18448 leader_election.cc:288] T d4c69dcb4423403bb96e9650ab867185 P 83a9e539db4444f697f95f180c02ddb9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28f59bbd6ffb4986b78211be1bcd85f6 (127.0.98.67:35699), cb855c3e9c83417aa24d7aee6043edd1 (127.0.98.65:41973)
I20260925 04:48:17.593796 18438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:17.590492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:54703 (local address 127.0.98.67:35699)
0925 04:48:17.590631 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:17.590636 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:17.590650 (+ 14us) server_negotiation.cc:407] Connection header received
0925 04:48:17.590713 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:17.590716 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:17.590763 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:17.590863 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:48:17.591279 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:17.591778 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:17.592848 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:17.593240 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:17.593359 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:17.593418 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:17.593604 (+ 186us) server_negotiation.cc:299] Negotiation successful
0925 04:48:17.593684 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 04:48:17.593796 18296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:17.590293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:35699 (local address 127.0.98.66:54703)
0925 04:48:17.590439 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:17.590457 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 04:48:17.590555 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:48:17.590967 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:48:17.590971 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:48:17.590990 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:48:17.591135 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:17.591143 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:17.591895 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:17.591900 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:17.592713 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:17.592722 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:17.593372 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:17.593376 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:17.593482 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:17.593500 (+ 18us) client_negotiation.cc:769] Sending connection context
0925 04:48:17.593591 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 04:48:17.593663 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260925 04:48:17.594069 18441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:17.590473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:55849 (local address 127.0.98.65:41973)
0925 04:48:17.590610 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:17.590615 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:17.591092 (+ 477us) server_negotiation.cc:407] Connection header received
0925 04:48:17.591206 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:17.591208 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:17.591252 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:17.591334 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:48:17.591632 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:17.592108 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:17.593114 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:17.593538 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:17.593651 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:17.593705 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:17.593884 (+ 179us) server_negotiation.cc:299] Negotiation successful
0925 04:48:17.593955 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 04:48:17.594089 18451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:17.590293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:41973 (local address 127.0.98.66:55849)
0925 04:48:17.590992 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:17.591005 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:48:17.591099 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:48:17.591348 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:48:17.591351 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:48:17.591368 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:48:17.591503 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:17.591509 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:17.592218 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:17.592223 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:17.592986 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:17.592995 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:17.593654 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:17.593658 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:17.593753 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:17.593779 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 04:48:17.593869 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 04:48:17.593945 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"mutex_wait_us":331,"thread_start_us":111,"threads_started":1}
I20260925 04:48:17.594547 18379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4c69dcb4423403bb96e9650ab867185" candidate_uuid: "83a9e539db4444f697f95f180c02ddb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28f59bbd6ffb4986b78211be1bcd85f6" is_pre_election: true
I20260925 04:48:17.594753 18117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4c69dcb4423403bb96e9650ab867185" candidate_uuid: "83a9e539db4444f697f95f180c02ddb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb855c3e9c83417aa24d7aee6043edd1" is_pre_election: true
I20260925 04:48:17.594852 18379 raft_consensus.cc:2454] T d4c69dcb4423403bb96e9650ab867185 P 28f59bbd6ffb4986b78211be1bcd85f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83a9e539db4444f697f95f180c02ddb9 in term 0.
I20260925 04:48:17.595048 18117 raft_consensus.cc:2454] T d4c69dcb4423403bb96e9650ab867185 P cb855c3e9c83417aa24d7aee6043edd1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83a9e539db4444f697f95f180c02ddb9 in term 0.
I20260925 04:48:17.595386 18182 leader_election.cc:302] T d4c69dcb4423403bb96e9650ab867185 P 83a9e539db4444f697f95f180c02ddb9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28f59bbd6ffb4986b78211be1bcd85f6, 83a9e539db4444f697f95f180c02ddb9; no voters:
I20260925 04:48:17.595685 18448 raft_consensus.cc:2781] T d4c69dcb4423403bb96e9650ab867185 P 83a9e539db4444f697f95f180c02ddb9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:48:17.595788 18448 raft_consensus.cc:482] T d4c69dcb4423403bb96e9650ab867185 P 83a9e539db4444f697f95f180c02ddb9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:48:17.595844 18448 raft_consensus.cc:3037] T d4c69dcb4423403bb96e9650ab867185 P 83a9e539db4444f697f95f180c02ddb9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:48:17.596870 18448 raft_consensus.cc:504] T d4c69dcb4423403bb96e9650ab867185 P 83a9e539db4444f697f95f180c02ddb9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83a9e539db4444f697f95f180c02ddb9" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 34313 } } peers { permanent_uuid: "28f59bbd6ffb4986b78211be1bcd85f6" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 35699 } } peers { permanent_uuid: "cb855c3e9c83417aa24d7aee6043edd1" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41973 } }
W20260925 04:48:17.596925 18164 tablet.cc:2369] T d4c69dcb4423403bb96e9650ab867185 P cb855c3e9c83417aa24d7aee6043edd1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:48:17.597258 18448 leader_election.cc:288] T d4c69dcb4423403bb96e9650ab867185 P 83a9e539db4444f697f95f180c02ddb9 [CANDIDATE]: Term 1 election: Requested vote from peers 28f59bbd6ffb4986b78211be1bcd85f6 (127.0.98.67:35699), cb855c3e9c83417aa24d7aee6043edd1 (127.0.98.65:41973)
I20260925 04:48:17.597606 18117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4c69dcb4423403bb96e9650ab867185" candidate_uuid: "83a9e539db4444f697f95f180c02ddb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb855c3e9c83417aa24d7aee6043edd1"
I20260925 04:48:17.597646 18379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4c69dcb4423403bb96e9650ab867185" candidate_uuid: "83a9e539db4444f697f95f180c02ddb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28f59bbd6ffb4986b78211be1bcd85f6"
I20260925 04:48:17.597744 18117 raft_consensus.cc:3037] T d4c69dcb4423403bb96e9650ab867185 P cb855c3e9c83417aa24d7aee6043edd1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:48:17.597787 18379 raft_consensus.cc:3037] T d4c69dcb4423403bb96e9650ab867185 P 28f59bbd6ffb4986b78211be1bcd85f6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:48:17.598932 18379 raft_consensus.cc:2454] T d4c69dcb4423403bb96e9650ab867185 P 28f59bbd6ffb4986b78211be1bcd85f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83a9e539db4444f697f95f180c02ddb9 in term 1.
I20260925 04:48:17.598991 18117 raft_consensus.cc:2454] T d4c69dcb4423403bb96e9650ab867185 P cb855c3e9c83417aa24d7aee6043edd1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83a9e539db4444f697f95f180c02ddb9 in term 1.
I20260925 04:48:17.599267 18182 leader_election.cc:302] T d4c69dcb4423403bb96e9650ab867185 P 83a9e539db4444f697f95f180c02ddb9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28f59bbd6ffb4986b78211be1bcd85f6, 83a9e539db4444f697f95f180c02ddb9; no voters:
I20260925 04:48:17.599452 18448 raft_consensus.cc:2781] T d4c69dcb4423403bb96e9650ab867185 P 83a9e539db4444f697f95f180c02ddb9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:48:17.599733 18448 raft_consensus.cc:686] T d4c69dcb4423403bb96e9650ab867185 P 83a9e539db4444f697f95f180c02ddb9 [term 1 LEADER]: Becoming Leader. State: Replica: 83a9e539db4444f697f95f180c02ddb9, State: Running, Role: LEADER
I20260925 04:48:17.600056 18448 consensus_queue.cc:237] T d4c69dcb4423403bb96e9650ab867185 P 83a9e539db4444f697f95f180c02ddb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83a9e539db4444f697f95f180c02ddb9" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 34313 } } peers { permanent_uuid: "28f59bbd6ffb4986b78211be1bcd85f6" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 35699 } } peers { permanent_uuid: "cb855c3e9c83417aa24d7aee6043edd1" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41973 } }
I20260925 04:48:17.603219 17975 catalog_manager.cc:5495] T d4c69dcb4423403bb96e9650ab867185 P 83a9e539db4444f697f95f180c02ddb9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 83a9e539db4444f697f95f180c02ddb9 (127.0.98.66). New cstate: current_term: 1 leader_uuid: "83a9e539db4444f697f95f180c02ddb9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83a9e539db4444f697f95f180c02ddb9" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 34313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "28f59bbd6ffb4986b78211be1bcd85f6" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 35699 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb855c3e9c83417aa24d7aee6043edd1" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41973 } health_report { overall_health: UNKNOWN } } }
I20260925 04:48:17.614252 18440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:17.610949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54774 (local address 127.0.98.66:34313)
0925 04:48:17.611078 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:17.611081 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:17.611094 (+ 13us) server_negotiation.cc:407] Connection header received
0925 04:48:17.611237 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:17.611240 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:17.611292 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:17.611376 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:48:17.611553 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:17.612025 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:17.612867 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:17.613070 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:17.613144 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:17.613421 (+ 277us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:48:17.613824 (+ 403us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:48:17.613914 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:17.614014 (+ 100us) server_negotiation.cc:299] Negotiation successful
0925 04:48:17.614061 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 04:48:17.620126 18117 raft_consensus.cc:1261] T d4c69dcb4423403bb96e9650ab867185 P cb855c3e9c83417aa24d7aee6043edd1 [term 1 FOLLOWER]: Refusing update from remote peer 83a9e539db4444f697f95f180c02ddb9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:48:17.620126 18379 raft_consensus.cc:1261] T d4c69dcb4423403bb96e9650ab867185 P 28f59bbd6ffb4986b78211be1bcd85f6 [term 1 FOLLOWER]: Refusing update from remote peer 83a9e539db4444f697f95f180c02ddb9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:48:17.620842 18452 consensus_queue.cc:1035] T d4c69dcb4423403bb96e9650ab867185 P 83a9e539db4444f697f95f180c02ddb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb855c3e9c83417aa24d7aee6043edd1" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:48:17.621038 18448 consensus_queue.cc:1035] T d4c69dcb4423403bb96e9650ab867185 P 83a9e539db4444f697f95f180c02ddb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28f59bbd6ffb4986b78211be1bcd85f6" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 35699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:48:17.629835 18457 mvcc.cc:205] Tried to move back new op lower bound from 7333116713443827712 to 7333116713372262400. Current Snapshot: MvccSnapshot[applied={T|T < 7333116713443827712}]
I20260925 04:48:17.636590 17974 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57382:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:48:17.636788 17974 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57382:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:48:17.639509 17974 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3d8b320ec056484991827ba1b1e13a5f: Sending DeleteTablet for 3 replicas of tablet d4c69dcb4423403bb96e9650ab867185
I20260925 04:48:17.640370 18359 tablet_service.cc:1514] Processing DeleteTablet for tablet d4c69dcb4423403bb96e9650ab867185 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:48:17 UTC) from {username='slave'} at 127.0.0.1:56208
I20260925 04:48:17.640549 18097 tablet_service.cc:1514] Processing DeleteTablet for tablet d4c69dcb4423403bb96e9650ab867185 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:48:17 UTC) from {username='slave'} at 127.0.0.1:45996
I20260925 04:48:17.640561 18228 tablet_service.cc:1514] Processing DeleteTablet for tablet d4c69dcb4423403bb96e9650ab867185 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:48:17 UTC) from {username='slave'} at 127.0.0.1:54764
I20260925 04:48:17.641024 18464 tablet_replica.cc:330] T d4c69dcb4423403bb96e9650ab867185 P 28f59bbd6ffb4986b78211be1bcd85f6: stopping tablet replica
I20260925 04:48:17.641026 18465 tablet_replica.cc:330] T d4c69dcb4423403bb96e9650ab867185 P cb855c3e9c83417aa24d7aee6043edd1: stopping tablet replica
I20260925 04:48:17.641309 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 18035
I20260925 04:48:17.641382 18464 raft_consensus.cc:2229] T d4c69dcb4423403bb96e9650ab867185 P 28f59bbd6ffb4986b78211be1bcd85f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:48:17.641525 18465 raft_consensus.cc:2229] T d4c69dcb4423403bb96e9650ab867185 P cb855c3e9c83417aa24d7aee6043edd1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:48:17.641710 18464 raft_consensus.cc:2258] T d4c69dcb4423403bb96e9650ab867185 P 28f59bbd6ffb4986b78211be1bcd85f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:48:17.644526 18464 ts_tablet_manager.cc:1905] T d4c69dcb4423403bb96e9650ab867185 P 28f59bbd6ffb4986b78211be1bcd85f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:48:17.646507 18466 tablet_replica.cc:330] T d4c69dcb4423403bb96e9650ab867185 P 83a9e539db4444f697f95f180c02ddb9: stopping tablet replica
I20260925 04:48:17.646878 18466 raft_consensus.cc:2229] T d4c69dcb4423403bb96e9650ab867185 P 83a9e539db4444f697f95f180c02ddb9 [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:48:17.647428 18466 raft_consensus.cc:2258] T d4c69dcb4423403bb96e9650ab867185 P 83a9e539db4444f697f95f180c02ddb9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:48:17.648111 18464 ts_tablet_manager.cc:1918] T d4c69dcb4423403bb96e9650ab867185 P 28f59bbd6ffb4986b78211be1bcd85f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:48:17.648222 18464 log.cc:1196] T d4c69dcb4423403bb96e9650ab867185 P 28f59bbd6ffb4986b78211be1bcd85f6: Deleting WAL directory at /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790311449873301-393-0/minicluster-data/ts-2/wal/wals/d4c69dcb4423403bb96e9650ab867185
I20260925 04:48:17.648548 18464 ts_tablet_manager.cc:1939] T d4c69dcb4423403bb96e9650ab867185 P 28f59bbd6ffb4986b78211be1bcd85f6: Deleting consensus metadata
I20260925 04:48:17.649405 18466 ts_tablet_manager.cc:1905] T d4c69dcb4423403bb96e9650ab867185 P 83a9e539db4444f697f95f180c02ddb9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:48:17.649880 17960 catalog_manager.cc:4841] TS 28f59bbd6ffb4986b78211be1bcd85f6 (127.0.98.67:35699): tablet d4c69dcb4423403bb96e9650ab867185 (table test-table [id=6e9354cde23a4c7b872936cd06aa01eb]) successfully deleted
W20260925 04:48:17.650218 17959 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.65:41973 (error 108)
I20260925 04:48:17.651306 18437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:17.650985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:41973 (local address 127.0.0.1:46002)
0925 04:48:17.651113 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:17.651201 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.65:41973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260925 04:48:17.651418 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 18166
W20260925 04:48:17.652168 17959 catalog_manager.cc:4584] TS cb855c3e9c83417aa24d7aee6043edd1 (127.0.98.65:41973): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d4c69dcb4423403bb96e9650ab867185: Network error: Client connection negotiation failed: client connection to 127.0.98.65:41973: connect: Connection refused (error 111)
I20260925 04:48:17.657866 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 18297
I20260925 04:48:17.658742 18437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:17.658022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:34313 (local address 127.0.0.1:54780)
0925 04:48:17.658562 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:17.658625 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.66:34313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442}
I20260925 04:48:17.663944 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 17944
2026-09-25T04:48:17Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1036 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:48:17 dist-test-slave-kghz krb5kdc[18470](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:48:17 dist-test-slave-kghz krb5kdc[18470](info): set up 2 sockets
Sep 25 04:48:17 dist-test-slave-kghz krb5kdc[18470](info): commencing operation
krb5kdc: starting...
W20260925 04:48:18.702189 393 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/krb5kdc/test-user.keytab.
Sep 25 04:48:18 dist-test-slave-kghz krb5kdc[18470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311698, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:48:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:48:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260925 04:48:18.825881 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:36729
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:39483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:36729
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:48:18.935559 18483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:48:18.935861 18483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:48:18.935940 18483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:48:18.939975 18483 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:48:18.940063 18483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:48:18.940100 18483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:48:18.940158 18483 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:48:18.940203 18483 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:48:18.945586 18483 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:39483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:36729
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:36729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:48:18.946970 18483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:48:18.948215 18483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 04:48:18.957145 18483 server_base.cc:1034] running on GCE node
W20260925 04:48:18.957086 18488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:48:18.957093 18489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:48:18.957090 18491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:48:18.958407 18483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:48:18.959538 18483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:48:18.960731 18483 hybrid_clock.cc:648] HybridClock initialized: now 1790311698960717 us; error 26 us; skew 500 ppm
Sep 25 04:48:18 dist-test-slave-kghz krb5kdc[18470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311698, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:48:18.979043 18483 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260925 04:48:18.980042 18483 webserver.cc:460] Webserver started at http://127.0.98.126:35277/ using document root <none> and password file <none>
I20260925 04:48:18.980669 18483 fs_manager.cc:359] Metadata directory not provided
I20260925 04:48:18.980769 18483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:48:18.981037 18483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:48:18.983294 18483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/minicluster-data/master-0/data/instance:
uuid: "91d38bf70a3b4df69fb6d49baef5ad53"
format_stamp: "Formatted at 2026-09-25 04:48:18 on dist-test-slave-kghz"
server_key: "f2473005c399da12040681e753c70329"
server_key_iv: "76a2c8e201f49d8336a75eb4fd85bb1d"
server_key_version: "encryptionkey@0"
I20260925 04:48:18.983886 18483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/minicluster-data/master-0/wal/instance:
uuid: "91d38bf70a3b4df69fb6d49baef5ad53"
format_stamp: "Formatted at 2026-09-25 04:48:18 on dist-test-slave-kghz"
server_key: "f2473005c399da12040681e753c70329"
server_key_iv: "76a2c8e201f49d8336a75eb4fd85bb1d"
server_key_version: "encryptionkey@0"
I20260925 04:48:18.987787 18483 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 04:48:18.990360 18498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:48:18.991786 18483 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 04:48:18.991950 18483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/minicluster-data/master-0/wal
uuid: "91d38bf70a3b4df69fb6d49baef5ad53"
format_stamp: "Formatted at 2026-09-25 04:48:18 on dist-test-slave-kghz"
server_key: "f2473005c399da12040681e753c70329"
server_key_iv: "76a2c8e201f49d8336a75eb4fd85bb1d"
server_key_version: "encryptionkey@0"
I20260925 04:48:18.992071 18483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:48:19.012838 18483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:48:19.016220 18483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:48:19.016435 18483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:48:19.023699 18483 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:36729
I20260925 04:48:19.023715 18550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:36729 every 8 connection(s)
I20260925 04:48:19.024641 18483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
I20260925 04:48:19.027350 18551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:48:19.032622 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 18483
I20260925 04:48:19.032759 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/minicluster-data/master-0/wal/instance
I20260925 04:48:19.032915 18551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91d38bf70a3b4df69fb6d49baef5ad53: Bootstrap starting.
I20260925 04:48:19.033064 393 external_mini_cluster.cc:1428] Setting key d86d1a2fe9b3f0382e2cabcd79ed2903
I20260925 04:48:19.035506 18551 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91d38bf70a3b4df69fb6d49baef5ad53: Neither blocks nor log segments found. Creating new log.
I20260925 04:48:19.036294 18551 log.cc:824] T 00000000000000000000000000000000 P 91d38bf70a3b4df69fb6d49baef5ad53: Log is configured to *not* fsync() on all Append() calls
I20260925 04:48:19.038201 18551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91d38bf70a3b4df69fb6d49baef5ad53: No bootstrap required, opened a new log
Sep 25 04:48:19 dist-test-slave-kghz krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311698, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:48:19.042749 18551 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91d38bf70a3b4df69fb6d49baef5ad53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91d38bf70a3b4df69fb6d49baef5ad53" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 36729 } }
I20260925 04:48:19.043044 18551 raft_consensus.cc:374] T 00000000000000000000000000000000 P 91d38bf70a3b4df69fb6d49baef5ad53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:48:19.043104 18551 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91d38bf70a3b4df69fb6d49baef5ad53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d38bf70a3b4df69fb6d49baef5ad53, State: Initialized, Role: FOLLOWER
I20260925 04:48:19.043633 18551 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91d38bf70a3b4df69fb6d49baef5ad53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91d38bf70a3b4df69fb6d49baef5ad53" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 36729 } }
I20260925 04:48:19.043818 18551 raft_consensus.cc:388] T 00000000000000000000000000000000 P 91d38bf70a3b4df69fb6d49baef5ad53 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:48:19.043900 18551 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91d38bf70a3b4df69fb6d49baef5ad53 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:48:19.044006 18551 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91d38bf70a3b4df69fb6d49baef5ad53 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:48:19.045158 18551 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91d38bf70a3b4df69fb6d49baef5ad53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91d38bf70a3b4df69fb6d49baef5ad53" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 36729 } }
I20260925 04:48:19.045521 18551 leader_election.cc:302] T 00000000000000000000000000000000 P 91d38bf70a3b4df69fb6d49baef5ad53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 91d38bf70a3b4df69fb6d49baef5ad53; no voters:
I20260925 04:48:19.045889 18551 leader_election.cc:288] T 00000000000000000000000000000000 P 91d38bf70a3b4df69fb6d49baef5ad53 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:48:19.046016 18556 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 91d38bf70a3b4df69fb6d49baef5ad53 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:48:19.046277 18556 raft_consensus.cc:686] T 00000000000000000000000000000000 P 91d38bf70a3b4df69fb6d49baef5ad53 [term 1 LEADER]: Becoming Leader. State: Replica: 91d38bf70a3b4df69fb6d49baef5ad53, State: Running, Role: LEADER
I20260925 04:48:19.046663 18556 consensus_queue.cc:237] T 00000000000000000000000000000000 P 91d38bf70a3b4df69fb6d49baef5ad53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91d38bf70a3b4df69fb6d49baef5ad53" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 36729 } }
I20260925 04:48:19.047159 18551 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91d38bf70a3b4df69fb6d49baef5ad53 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:48:19.048364 18557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91d38bf70a3b4df69fb6d49baef5ad53 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "91d38bf70a3b4df69fb6d49baef5ad53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91d38bf70a3b4df69fb6d49baef5ad53" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 36729 } } }
I20260925 04:48:19.048518 18557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91d38bf70a3b4df69fb6d49baef5ad53 [sys.catalog]: This master's current role is: LEADER
I20260925 04:48:19.048866 18558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91d38bf70a3b4df69fb6d49baef5ad53 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91d38bf70a3b4df69fb6d49baef5ad53. Latest consensus state: current_term: 1 leader_uuid: "91d38bf70a3b4df69fb6d49baef5ad53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91d38bf70a3b4df69fb6d49baef5ad53" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 36729 } } }
I20260925 04:48:19.048885 18554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:19.034464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57664 (local address 127.0.98.126:36729)
0925 04:48:19.034871 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:19.034893 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:19.034951 (+ 58us) server_negotiation.cc:407] Connection header received
0925 04:48:19.035666 (+ 715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:19.035685 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:19.036029 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:19.036309 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:19.037194 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:19.038186 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:19.039422 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:19.039884 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:19.040013 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:19.042928 (+ 2915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:19.042934 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:19.042950 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:48:19.042981 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:19.045136 (+ 2155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:19.045639 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:19.045644 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:19.045650 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:19.045787 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:19.046171 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:19.046177 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:19.046181 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:19.046745 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:19.046971 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:19.047407 (+ 436us) server_negotiation.cc:299] Negotiation successful
0925 04:48:19.047818 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":136,"threads_started":1}
I20260925 04:48:19.048990 18558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91d38bf70a3b4df69fb6d49baef5ad53 [sys.catalog]: This master's current role is: LEADER
W20260925 04:48:19.052481 18571 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 91d38bf70a3b4df69fb6d49baef5ad53: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 04:48:19.052567 18571 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 04:48:19.052745 18565 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:48:19.053642 18565 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:48:19.059214 18565 catalog_manager.cc:1347] Generated new cluster ID: f18cc81986b14fc0ab510f9e9e380b03
I20260925 04:48:19.059283 18565 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:48:19.070978 18565 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:48:19.071973 18565 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:48:19.079936 18565 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 91d38bf70a3b4df69fb6d49baef5ad53: Generated new TSK 0
I20260925 04:48:19.080646 18565 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/krb5kdc/kudu_127.0.98.65.keytab.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/krb5kdc/kudu_127.0.98.65.keytab.
I20260925 04:48:19.122027 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:0
--local_ip_for_outbound_sockets=127.0.98.65
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:36729
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:39483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:48:19.238224 18577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:48:19.238493 18577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:48:19.238536 18577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:48:19.242525 18577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:48:19.242583 18577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:48:19.242641 18577 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260925 04:48:19.248287 18577 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:39483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:36729
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18577
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:48:19.249338 18577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:48:19.250370 18577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:48:19.260097 18582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:48:19.260136 18585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:48:19.260136 18583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:48:19.260341 18577 server_base.cc:1034] running on GCE node
I20260925 04:48:19.260776 18577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:48:19.261488 18577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:48:19.262645 18577 hybrid_clock.cc:648] HybridClock initialized: now 1790311699262631 us; error 34 us; skew 500 ppm
Sep 25 04:48:19 dist-test-slave-kghz krb5kdc[18470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311699, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:48:19.265435 18577 init.cc:377] Logged in from keytab as kudu/127.0.98.65@KRBTEST.COM (short username kudu)
I20260925 04:48:19.266404 18577 webserver.cc:460] Webserver started at http://127.0.98.65:37247/ using document root <none> and password file <none>
I20260925 04:48:19.266968 18577 fs_manager.cc:359] Metadata directory not provided
I20260925 04:48:19.267058 18577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:48:19.267293 18577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:48:19.269361 18577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/minicluster-data/ts-0/data/instance:
uuid: "26c6811dc2bc42fabfa7a3579aaec754"
format_stamp: "Formatted at 2026-09-25 04:48:19 on dist-test-slave-kghz"
server_key: "aa3275f086b9a1df6ca4bebbb125e69d"
server_key_iv: "a5183e1dabf9eabb4dc908654ff4cc82"
server_key_version: "encryptionkey@0"
I20260925 04:48:19.269948 18577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/minicluster-data/ts-0/wal/instance:
uuid: "26c6811dc2bc42fabfa7a3579aaec754"
format_stamp: "Formatted at 2026-09-25 04:48:19 on dist-test-slave-kghz"
server_key: "aa3275f086b9a1df6ca4bebbb125e69d"
server_key_iv: "a5183e1dabf9eabb4dc908654ff4cc82"
server_key_version: "encryptionkey@0"
I20260925 04:48:19.273792 18577 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260925 04:48:19.276401 18592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:48:19.277843 18577 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 04:48:19.277987 18577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/minicluster-data/ts-0/wal
uuid: "26c6811dc2bc42fabfa7a3579aaec754"
format_stamp: "Formatted at 2026-09-25 04:48:19 on dist-test-slave-kghz"
server_key: "aa3275f086b9a1df6ca4bebbb125e69d"
server_key_iv: "a5183e1dabf9eabb4dc908654ff4cc82"
server_key_version: "encryptionkey@0"
I20260925 04:48:19.278108 18577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:48:19.292563 18577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:48:19.296060 18577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:48:19.296360 18577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:48:19.296988 18577 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:48:19.297855 18577 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:48:19.297932 18577 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:48:19.298009 18577 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:48:19.298053 18577 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:48:19.307739 18577 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:33343
I20260925 04:48:19.307761 18705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:33343 every 8 connection(s)
I20260925 04:48:19.308678 18577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
I20260925 04:48:19.309114 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 18577
I20260925 04:48:19.309245 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/minicluster-data/ts-0/wal/instance
I20260925 04:48:19.309587 393 external_mini_cluster.cc:1428] Setting key 80185fdaac938bf5468e94919b0fccb7
WARNING: no policy specified for kudu/127.0.98.66@KRBTEST.COM; defaulting to no policy
Sep 25 04:48:19 dist-test-slave-kghz krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311699, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.66@KRBTEST.COM" created.
I20260925 04:48:19.323836 18554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:19.310541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:39869 (local address 127.0.98.126:36729)
0925 04:48:19.310690 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:19.310695 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:19.311822 (+ 1127us) server_negotiation.cc:407] Connection header received
0925 04:48:19.313396 (+ 1574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:19.313399 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:19.313446 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:19.313519 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:19.315173 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:19.315769 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:19.317010 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:19.317417 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:19.317544 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:19.320062 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:19.320065 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:19.320068 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:48:19.320092 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:19.321679 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:19.322302 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:19.322307 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:19.322309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:19.322401 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:19.322750 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:19.322753 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:19.322755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:19.322940 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:19.323046 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:19.323541 (+ 495us) server_negotiation.cc:299] Negotiation successful
0925 04:48:19.323667 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 04:48:19.324664 18708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:19.311072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:36729 (local address 127.0.98.65:39869)
0925 04:48:19.311648 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:19.311703 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 04:48:19.313149 (+ 1446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:48:19.313718 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:48:19.313729 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:48:19.314301 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:48:19.315017 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:19.315040 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:19.315911 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:19.315915 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:19.316876 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:19.316886 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:19.317565 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:19.317568 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:19.317726 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:19.318428 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:48:19.318444 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:48:19.319896 (+ 1452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:48:19.321839 (+ 1943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:48:19.321844 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:48:19.321856 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:48:19.322154 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:48:19.322494 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:48:19.322501 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:48:19.322503 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:48:19.322644 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:48:19.323045 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:48:19.323052 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:48:19.323299 (+ 247us) client_negotiation.cc:769] Sending connection context
0925 04:48:19.323499 (+ 200us) client_negotiation.cc:240] Negotiation successful
0925 04:48:19.323836 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":143,"threads_started":1}
I20260925 04:48:19.326159 18706 heartbeater.cc:347] Connected to a master server at 127.0.98.126:36729
I20260925 04:48:19.326459 18706 heartbeater.cc:464] Registering TS with master...
I20260925 04:48:19.326951 18706 heartbeater.cc:511] Master 127.0.98.126:36729 requested a full tablet report, sending...
I20260925 04:48:19.328428 18515 ts_manager.cc:194] Registered new tserver with Master: 26c6811dc2bc42fabfa7a3579aaec754 (127.0.98.65:33343)
I20260925 04:48:19.329569 18515 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.65@KRBTEST.COM'} at 127.0.98.65:39869
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/krb5kdc/kudu_127.0.98.66.keytab.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/krb5kdc/kudu_127.0.98.66.keytab.
I20260925 04:48:19.337692 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:0
--local_ip_for_outbound_sockets=127.0.98.66
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:36729
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:39483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:48:19.448859 18711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:48:19.449118 18711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:48:19.449162 18711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:48:19.453198 18711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:48:19.453253 18711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:48:19.453312 18711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260925 04:48:19.458484 18711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:39483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:36729
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:48:19.459435 18711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:48:19.460400 18711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:48:19.470340 18719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:48:19.470340 18716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:48:19.470340 18717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:48:19.470376 18711 server_base.cc:1034] running on GCE node
I20260925 04:48:19.470927 18711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:48:19.471560 18711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:48:19.472712 18711 hybrid_clock.cc:648] HybridClock initialized: now 1790311699472697 us; error 26 us; skew 500 ppm
Sep 25 04:48:19 dist-test-slave-kghz krb5kdc[18470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311699, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:48:19.475524 18711 init.cc:377] Logged in from keytab as kudu/127.0.98.66@KRBTEST.COM (short username kudu)
I20260925 04:48:19.476399 18711 webserver.cc:460] Webserver started at http://127.0.98.66:36705/ using document root <none> and password file <none>
I20260925 04:48:19.476956 18711 fs_manager.cc:359] Metadata directory not provided
I20260925 04:48:19.477042 18711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:48:19.477268 18711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:48:19.479343 18711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/minicluster-data/ts-1/data/instance:
uuid: "7cd2a1e3c7114fb68fe20199e8dd3782"
format_stamp: "Formatted at 2026-09-25 04:48:19 on dist-test-slave-kghz"
server_key: "d4084625e60aa1bcdf463a8cd771f283"
server_key_iv: "12994280ccf7709c0236a3f8918bad99"
server_key_version: "encryptionkey@0"
I20260925 04:48:19.479892 18711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/minicluster-data/ts-1/wal/instance:
uuid: "7cd2a1e3c7114fb68fe20199e8dd3782"
format_stamp: "Formatted at 2026-09-25 04:48:19 on dist-test-slave-kghz"
server_key: "d4084625e60aa1bcdf463a8cd771f283"
server_key_iv: "12994280ccf7709c0236a3f8918bad99"
server_key_version: "encryptionkey@0"
I20260925 04:48:19.483747 18711 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 04:48:19.486429 18726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:48:19.487743 18711 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:48:19.487937 18711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/minicluster-data/ts-1/wal
uuid: "7cd2a1e3c7114fb68fe20199e8dd3782"
format_stamp: "Formatted at 2026-09-25 04:48:19 on dist-test-slave-kghz"
server_key: "d4084625e60aa1bcdf463a8cd771f283"
server_key_iv: "12994280ccf7709c0236a3f8918bad99"
server_key_version: "encryptionkey@0"
I20260925 04:48:19.488060 18711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:48:19.502002 18711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:48:19.505317 18711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:48:19.505539 18711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:48:19.506158 18711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:48:19.507025 18711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:48:19.507102 18711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:48:19.507174 18711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:48:19.507225 18711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:48:19.516440 18711 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:37005
I20260925 04:48:19.516467 18839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:37005 every 8 connection(s)
I20260925 04:48:19.517318 18711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
I20260925 04:48:19.524109 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 18711
I20260925 04:48:19.524264 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/minicluster-data/ts-1/wal/instance
I20260925 04:48:19.524562 393 external_mini_cluster.cc:1428] Setting key fe226c0fcc208b96f56c10a6fd5bd8a9
Sep 25 04:48:19 dist-test-slave-kghz krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311699, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:48:19.530754 18554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:19.519339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:42471 (local address 127.0.98.126:36729)
0925 04:48:19.519497 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:19.519509 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:19.520355 (+ 846us) server_negotiation.cc:407] Connection header received
0925 04:48:19.521180 (+ 825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:19.521183 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:19.521229 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:19.521311 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:19.522554 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:19.523084 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:19.524242 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:19.524527 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:19.524643 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:19.527124 (+ 2481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:19.527128 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:19.527130 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:48:19.527162 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:19.528660 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:19.529183 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:19.529187 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:19.529189 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:19.529273 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:19.529616 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:19.529621 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:19.529622 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:19.529816 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:19.529927 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:19.530465 (+ 538us) server_negotiation.cc:299] Negotiation successful
0925 04:48:19.530589 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 04:48:19.531572 18842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:19.519735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:36729 (local address 127.0.98.66:42471)
0925 04:48:19.520177 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:19.520220 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 04:48:19.520982 (+ 762us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:48:19.521486 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:48:19.521492 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:48:19.521837 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:48:19.522383 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:19.522399 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:19.523190 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:19.523193 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:19.524111 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:19.524120 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:19.524678 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:19.524682 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:19.524855 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:19.525474 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:48:19.525490 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:48:19.526963 (+ 1473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:48:19.528791 (+ 1828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:48:19.528797 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:48:19.528811 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:48:19.529081 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:48:19.529374 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:48:19.529377 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:48:19.529379 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:48:19.529519 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:48:19.529923 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:48:19.529930 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:48:19.530175 (+ 245us) client_negotiation.cc:769] Sending connection context
0925 04:48:19.530420 (+ 245us) client_negotiation.cc:240] Negotiation successful
0925 04:48:19.530741 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":102,"threads_started":1}
I20260925 04:48:19.532819 18840 heartbeater.cc:347] Connected to a master server at 127.0.98.126:36729
I20260925 04:48:19.533067 18840 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.67@KRBTEST.COM; defaulting to no policy
I20260925 04:48:19.533669 18840 heartbeater.cc:511] Master 127.0.98.126:36729 requested a full tablet report, sending...
I20260925 04:48:19.534987 18515 ts_manager.cc:194] Registered new tserver with Master: 7cd2a1e3c7114fb68fe20199e8dd3782 (127.0.98.66:37005)
I20260925 04:48:19.535888 18515 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.66@KRBTEST.COM'} at 127.0.98.66:42471
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/krb5kdc/kudu_127.0.98.67.keytab.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/krb5kdc/kudu_127.0.98.67.keytab.
I20260925 04:48:19.553462 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:0
--local_ip_for_outbound_sockets=127.0.98.67
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:36729
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:39483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:48:19.666018 18845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:48:19.666385 18845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:48:19.666450 18845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:48:19.670620 18845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:48:19.670677 18845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:48:19.670737 18845 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260925 04:48:19.675858 18845 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:39483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:36729
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18845
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:48:19.676792 18845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:48:19.677788 18845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:48:19.687459 18850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:48:19.687464 18853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:48:19.687516 18845 server_base.cc:1034] running on GCE node
W20260925 04:48:19.687459 18851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:48:19.688223 18845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:48:19.688894 18845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:48:19.690109 18845 hybrid_clock.cc:648] HybridClock initialized: now 1790311699690079 us; error 52 us; skew 500 ppm
Sep 25 04:48:19 dist-test-slave-kghz krb5kdc[18470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311699, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:48:19.693146 18845 init.cc:377] Logged in from keytab as kudu/127.0.98.67@KRBTEST.COM (short username kudu)
I20260925 04:48:19.694077 18845 webserver.cc:460] Webserver started at http://127.0.98.67:40209/ using document root <none> and password file <none>
I20260925 04:48:19.694731 18845 fs_manager.cc:359] Metadata directory not provided
I20260925 04:48:19.694904 18845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:48:19.695159 18845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:48:19.697320 18845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/minicluster-data/ts-2/data/instance:
uuid: "622fa574e745488c9c18328a01285572"
format_stamp: "Formatted at 2026-09-25 04:48:19 on dist-test-slave-kghz"
server_key: "25aa0940a9be778705a4672c172e2555"
server_key_iv: "6bd8782bc469a605a4f457bd5969cdb7"
server_key_version: "encryptionkey@0"
I20260925 04:48:19.697908 18845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/minicluster-data/ts-2/wal/instance:
uuid: "622fa574e745488c9c18328a01285572"
format_stamp: "Formatted at 2026-09-25 04:48:19 on dist-test-slave-kghz"
server_key: "25aa0940a9be778705a4672c172e2555"
server_key_iv: "6bd8782bc469a605a4f457bd5969cdb7"
server_key_version: "encryptionkey@0"
I20260925 04:48:19.701857 18845 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 04:48:19.704635 18860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:48:19.705994 18845 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 04:48:19.706169 18845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/minicluster-data/ts-2/wal
uuid: "622fa574e745488c9c18328a01285572"
format_stamp: "Formatted at 2026-09-25 04:48:19 on dist-test-slave-kghz"
server_key: "25aa0940a9be778705a4672c172e2555"
server_key_iv: "6bd8782bc469a605a4f457bd5969cdb7"
server_key_version: "encryptionkey@0"
I20260925 04:48:19.706351 18845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:48:19.726727 18845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:48:19.729722 18845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:48:19.729931 18845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:48:19.730556 18845 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:48:19.731379 18845 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:48:19.731449 18845 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:48:19.731521 18845 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:48:19.731572 18845 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:48:19.740357 18845 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:45617
I20260925 04:48:19.740383 18973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:45617 every 8 connection(s)
I20260925 04:48:19.741267 18845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
I20260925 04:48:19.750041 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 18845
I20260925 04:48:19.750217 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/minicluster-data/ts-2/wal/instance
I20260925 04:48:19.750546 393 external_mini_cluster.cc:1428] Setting key 0f80236a83945dad2f8e4d063d040f7f
Sep 25 04:48:19 dist-test-slave-kghz krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311699, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:48:19.755745 18554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:19.743218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:33215 (local address 127.0.98.126:36729)
0925 04:48:19.743374 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:19.743379 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:19.744282 (+ 903us) server_negotiation.cc:407] Connection header received
0925 04:48:19.745544 (+ 1262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:19.745547 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:19.745595 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:19.745678 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:19.746924 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:19.747555 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:19.748789 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:19.748999 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:19.749081 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:19.751979 (+ 2898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:19.751984 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:19.751988 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:48:19.752018 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:19.753693 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:19.754333 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:19.754358 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:19.754359 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:19.754446 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:19.754787 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:19.754789 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:19.754790 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:19.754974 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:19.755051 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:19.755521 (+ 470us) server_negotiation.cc:299] Negotiation successful
0925 04:48:19.755619 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 04:48:19.756572 18976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:19.743662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:36729 (local address 127.0.98.67:33215)
0925 04:48:19.744121 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:19.744166 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 04:48:19.745309 (+ 1143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:48:19.745845 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:48:19.745852 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:48:19.746224 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:48:19.746783 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:19.746797 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:19.747669 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:19.747672 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:19.748684 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:19.748694 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:19.749128 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:19.749130 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:19.749269 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:19.749978 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:48:19.749995 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:48:19.751753 (+ 1758us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:48:19.753831 (+ 2078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:48:19.753836 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:48:19.753848 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:48:19.754132 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:48:19.754536 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:48:19.754539 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:48:19.754540 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:48:19.754679 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:48:19.755063 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:48:19.755069 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:48:19.755304 (+ 235us) client_negotiation.cc:769] Sending connection context
0925 04:48:19.755470 (+ 166us) client_negotiation.cc:240] Negotiation successful
0925 04:48:19.755727 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":103,"threads_started":1}
I20260925 04:48:19.757733 18974 heartbeater.cc:347] Connected to a master server at 127.0.98.126:36729
I20260925 04:48:19.757946 18974 heartbeater.cc:464] Registering TS with master...
I20260925 04:48:19.758463 18974 heartbeater.cc:511] Master 127.0.98.126:36729 requested a full tablet report, sending...
I20260925 04:48:19.759469 18515 ts_manager.cc:194] Registered new tserver with Master: 622fa574e745488c9c18328a01285572 (127.0.98.67:45617)
I20260925 04:48:19.760004 18515 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.67@KRBTEST.COM'} at 127.0.98.67:33215
I20260925 04:48:19.764657 393 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 04:48:19.774029 18554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:19.766738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34526 (local address 127.0.98.126:36729)
0925 04:48:19.766872 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:19.766875 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:19.766991 (+ 116us) server_negotiation.cc:407] Connection header received
0925 04:48:19.767062 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:19.767065 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:19.767111 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:19.767187 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:19.767851 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:19.768468 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:19.769269 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:19.769478 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:19.769566 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:19.771117 (+ 1551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:19.771120 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:19.771123 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:48:19.771153 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:19.772495 (+ 1342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:19.772962 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:19.772965 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:19.772966 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:19.773041 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:19.773311 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:19.773313 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:19.773314 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:19.773508 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:19.773620 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:19.773836 (+ 216us) server_negotiation.cc:299] Negotiation successful
0925 04:48:19.773911 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 04:48:19.776957 18515 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34526:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:48:19.780180 18515 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:48:19.793964 18985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:19.788200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:37005 (local address 127.0.0.1:55186)
0925 04:48:19.788630 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:19.788683 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 04:48:19.788819 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:48:19.789416 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:48:19.789422 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:48:19.789452 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:48:19.789807 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:19.789816 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:19.791210 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:19.791215 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:19.792536 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:19.792544 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:19.793392 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:19.793395 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:19.793501 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:19.793542 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 04:48:19.793663 (+ 121us) client_negotiation.cc:240] Negotiation successful
0925 04:48:19.793788 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"mutex_wait_us":220,"thread_start_us":74,"threads_started":1}
I20260925 04:48:19.793974 18987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:19.788332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55186 (local address 127.0.98.66:37005)
0925 04:48:19.788790 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:19.788802 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:19.788828 (+ 26us) server_negotiation.cc:407] Connection header received
0925 04:48:19.789023 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:19.789029 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:19.789245 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:19.789422 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:48:19.789966 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:19.791011 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:19.792663 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:19.793255 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:19.793386 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:19.793551 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:19.793709 (+ 158us) server_negotiation.cc:299] Negotiation successful
0925 04:48:19.793806 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":131,"threads_started":1}
I20260925 04:48:19.794785 18986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:19.788225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:45617 (local address 127.0.0.1:42152)
0925 04:48:19.789407 (+ 1182us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:19.789420 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:48:19.789503 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:48:19.790410 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:48:19.790414 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:48:19.790433 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:48:19.790516 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:19.790522 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:19.791952 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:19.791956 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:19.793194 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:19.793204 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:19.794400 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:19.794404 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:19.794497 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:19.794518 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 04:48:19.794578 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 04:48:19.794651 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1118,"mutex_wait_us":1,"thread_start_us":93,"threads_started":1}
I20260925 04:48:19.795610 18988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:19.788911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:33343 (local address 127.0.0.1:60672)
0925 04:48:19.789730 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:19.789743 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:48:19.789839 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:48:19.790516 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:48:19.790519 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:48:19.790535 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:48:19.790648 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:19.790654 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:19.792096 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:19.792099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:19.793261 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:19.793270 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:19.795196 (+ 1926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:19.795199 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:19.795291 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:19.795312 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 04:48:19.795411 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 04:48:19.795481 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":747,"thread_start_us":75,"threads_started":1}
I20260925 04:48:19.795748 18989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:19.789083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60672 (local address 127.0.98.65:33343)
0925 04:48:19.789658 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:19.789674 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:19.789933 (+ 259us) server_negotiation.cc:407] Connection header received
0925 04:48:19.790018 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:19.790023 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:19.790197 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:19.790365 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:48:19.790813 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:19.791967 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:19.793365 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:19.795073 (+ 1708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:19.795196 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:19.795336 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:19.795458 (+ 122us) server_negotiation.cc:299] Negotiation successful
0925 04:48:19.795598 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":70,"threads_started":1}
I20260925 04:48:19.796408 18990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:19.789007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42152 (local address 127.0.98.67:45617)
0925 04:48:19.789955 (+ 948us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:19.789969 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:19.789987 (+ 18us) server_negotiation.cc:407] Connection header received
0925 04:48:19.790051 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:19.790055 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:19.790197 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:19.790393 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:48:19.790635 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:19.791772 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:19.793331 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:19.794231 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:19.794388 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:19.796094 (+ 1706us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:19.796145 (+ 51us) server_negotiation.cc:299] Negotiation successful
0925 04:48:19.796246 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":845,"thread_start_us":160,"threads_started":1}
I20260925 04:48:19.797767 18774 tablet_service.cc:1467] Processing CreateTablet for tablet 57f328136acc438bb0fefd739b67612e (DEFAULT_TABLE table=test-table [id=16d2e57b736d43d88317fab2996d59d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:48:19.798570 18631 tablet_service.cc:1467] Processing CreateTablet for tablet 57f328136acc438bb0fefd739b67612e (DEFAULT_TABLE table=test-table [id=16d2e57b736d43d88317fab2996d59d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:48:19.798959 18774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57f328136acc438bb0fefd739b67612e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:48:19.799393 18631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57f328136acc438bb0fefd739b67612e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:48:19.799057 18908 tablet_service.cc:1467] Processing CreateTablet for tablet 57f328136acc438bb0fefd739b67612e (DEFAULT_TABLE table=test-table [id=16d2e57b736d43d88317fab2996d59d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:48:19.800000 18908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57f328136acc438bb0fefd739b67612e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:48:19.805403 18991 tablet_bootstrap.cc:492] T 57f328136acc438bb0fefd739b67612e P 26c6811dc2bc42fabfa7a3579aaec754: Bootstrap starting.
I20260925 04:48:19.806730 18992 tablet_bootstrap.cc:492] T 57f328136acc438bb0fefd739b67612e P 7cd2a1e3c7114fb68fe20199e8dd3782: Bootstrap starting.
I20260925 04:48:19.807808 18991 tablet_bootstrap.cc:654] T 57f328136acc438bb0fefd739b67612e P 26c6811dc2bc42fabfa7a3579aaec754: Neither blocks nor log segments found. Creating new log.
I20260925 04:48:19.809077 18991 log.cc:824] T 57f328136acc438bb0fefd739b67612e P 26c6811dc2bc42fabfa7a3579aaec754: Log is configured to *not* fsync() on all Append() calls
I20260925 04:48:19.809317 18992 tablet_bootstrap.cc:654] T 57f328136acc438bb0fefd739b67612e P 7cd2a1e3c7114fb68fe20199e8dd3782: Neither blocks nor log segments found. Creating new log.
I20260925 04:48:19.809885 18993 tablet_bootstrap.cc:492] T 57f328136acc438bb0fefd739b67612e P 622fa574e745488c9c18328a01285572: Bootstrap starting.
I20260925 04:48:19.810150 18992 log.cc:824] T 57f328136acc438bb0fefd739b67612e P 7cd2a1e3c7114fb68fe20199e8dd3782: Log is configured to *not* fsync() on all Append() calls
I20260925 04:48:19.811698 18991 tablet_bootstrap.cc:492] T 57f328136acc438bb0fefd739b67612e P 26c6811dc2bc42fabfa7a3579aaec754: No bootstrap required, opened a new log
I20260925 04:48:19.811918 18991 ts_tablet_manager.cc:1397] T 57f328136acc438bb0fefd739b67612e P 26c6811dc2bc42fabfa7a3579aaec754: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20260925 04:48:19.811956 18993 tablet_bootstrap.cc:654] T 57f328136acc438bb0fefd739b67612e P 622fa574e745488c9c18328a01285572: Neither blocks nor log segments found. Creating new log.
I20260925 04:48:19.812186 18992 tablet_bootstrap.cc:492] T 57f328136acc438bb0fefd739b67612e P 7cd2a1e3c7114fb68fe20199e8dd3782: No bootstrap required, opened a new log
I20260925 04:48:19.812362 18992 ts_tablet_manager.cc:1397] T 57f328136acc438bb0fefd739b67612e P 7cd2a1e3c7114fb68fe20199e8dd3782: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 04:48:19.813207 18993 log.cc:824] T 57f328136acc438bb0fefd739b67612e P 622fa574e745488c9c18328a01285572: Log is configured to *not* fsync() on all Append() calls
I20260925 04:48:19.814875 18993 tablet_bootstrap.cc:492] T 57f328136acc438bb0fefd739b67612e P 622fa574e745488c9c18328a01285572: No bootstrap required, opened a new log
I20260925 04:48:19.815063 18993 ts_tablet_manager.cc:1397] T 57f328136acc438bb0fefd739b67612e P 622fa574e745488c9c18328a01285572: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 04:48:19.816026 18992 raft_consensus.cc:348] T 57f328136acc438bb0fefd739b67612e P 7cd2a1e3c7114fb68fe20199e8dd3782 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cd2a1e3c7114fb68fe20199e8dd3782" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37005 } } peers { permanent_uuid: "622fa574e745488c9c18328a01285572" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 45617 } } peers { permanent_uuid: "26c6811dc2bc42fabfa7a3579aaec754" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33343 } }
I20260925 04:48:19.815987 18991 raft_consensus.cc:348] T 57f328136acc438bb0fefd739b67612e P 26c6811dc2bc42fabfa7a3579aaec754 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cd2a1e3c7114fb68fe20199e8dd3782" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37005 } } peers { permanent_uuid: "622fa574e745488c9c18328a01285572" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 45617 } } peers { permanent_uuid: "26c6811dc2bc42fabfa7a3579aaec754" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33343 } }
I20260925 04:48:19.816270 18992 raft_consensus.cc:374] T 57f328136acc438bb0fefd739b67612e P 7cd2a1e3c7114fb68fe20199e8dd3782 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:48:19.816270 18991 raft_consensus.cc:374] T 57f328136acc438bb0fefd739b67612e P 26c6811dc2bc42fabfa7a3579aaec754 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:48:19.816330 18991 raft_consensus.cc:729] T 57f328136acc438bb0fefd739b67612e P 26c6811dc2bc42fabfa7a3579aaec754 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26c6811dc2bc42fabfa7a3579aaec754, State: Initialized, Role: FOLLOWER
I20260925 04:48:19.816365 18992 raft_consensus.cc:729] T 57f328136acc438bb0fefd739b67612e P 7cd2a1e3c7114fb68fe20199e8dd3782 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cd2a1e3c7114fb68fe20199e8dd3782, State: Initialized, Role: FOLLOWER
I20260925 04:48:19.816834 18992 consensus_queue.cc:260] T 57f328136acc438bb0fefd739b67612e P 7cd2a1e3c7114fb68fe20199e8dd3782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cd2a1e3c7114fb68fe20199e8dd3782" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37005 } } peers { permanent_uuid: "622fa574e745488c9c18328a01285572" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 45617 } } peers { permanent_uuid: "26c6811dc2bc42fabfa7a3579aaec754" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33343 } }
I20260925 04:48:19.816834 18991 consensus_queue.cc:260] T 57f328136acc438bb0fefd739b67612e P 26c6811dc2bc42fabfa7a3579aaec754 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cd2a1e3c7114fb68fe20199e8dd3782" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37005 } } peers { permanent_uuid: "622fa574e745488c9c18328a01285572" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 45617 } } peers { permanent_uuid: "26c6811dc2bc42fabfa7a3579aaec754" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33343 } }
I20260925 04:48:19.817526 18706 heartbeater.cc:503] Master 127.0.98.126:36729 was elected leader, sending a full tablet report...
I20260925 04:48:19.817890 18991 ts_tablet_manager.cc:1428] T 57f328136acc438bb0fefd739b67612e P 26c6811dc2bc42fabfa7a3579aaec754: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 04:48:19.818397 18993 raft_consensus.cc:348] T 57f328136acc438bb0fefd739b67612e P 622fa574e745488c9c18328a01285572 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cd2a1e3c7114fb68fe20199e8dd3782" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37005 } } peers { permanent_uuid: "622fa574e745488c9c18328a01285572" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 45617 } } peers { permanent_uuid: "26c6811dc2bc42fabfa7a3579aaec754" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33343 } }
I20260925 04:48:19.818722 18993 raft_consensus.cc:374] T 57f328136acc438bb0fefd739b67612e P 622fa574e745488c9c18328a01285572 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:48:19.818823 18992 ts_tablet_manager.cc:1428] T 57f328136acc438bb0fefd739b67612e P 7cd2a1e3c7114fb68fe20199e8dd3782: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 04:48:19.818938 18840 heartbeater.cc:503] Master 127.0.98.126:36729 was elected leader, sending a full tablet report...
I20260925 04:48:19.818815 18993 raft_consensus.cc:729] T 57f328136acc438bb0fefd739b67612e P 622fa574e745488c9c18328a01285572 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 622fa574e745488c9c18328a01285572, State: Initialized, Role: FOLLOWER
I20260925 04:48:19.820107 18993 consensus_queue.cc:260] T 57f328136acc438bb0fefd739b67612e P 622fa574e745488c9c18328a01285572 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cd2a1e3c7114fb68fe20199e8dd3782" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37005 } } peers { permanent_uuid: "622fa574e745488c9c18328a01285572" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 45617 } } peers { permanent_uuid: "26c6811dc2bc42fabfa7a3579aaec754" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33343 } }
I20260925 04:48:19.820796 18974 heartbeater.cc:503] Master 127.0.98.126:36729 was elected leader, sending a full tablet report...
I20260925 04:48:19.821094 18993 ts_tablet_manager.cc:1428] T 57f328136acc438bb0fefd739b67612e P 622fa574e745488c9c18328a01285572: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260925 04:48:19.992264 18975 tablet.cc:2369] T 57f328136acc438bb0fefd739b67612e P 622fa574e745488c9c18328a01285572: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:48:20.016666 18998 raft_consensus.cc:482] T 57f328136acc438bb0fefd739b67612e P 7cd2a1e3c7114fb68fe20199e8dd3782 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:48:20.016922 18998 raft_consensus.cc:504] T 57f328136acc438bb0fefd739b67612e P 7cd2a1e3c7114fb68fe20199e8dd3782 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cd2a1e3c7114fb68fe20199e8dd3782" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37005 } } peers { permanent_uuid: "622fa574e745488c9c18328a01285572" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 45617 } } peers { permanent_uuid: "26c6811dc2bc42fabfa7a3579aaec754" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33343 } }
I20260925 04:48:20.018211 18998 leader_election.cc:288] T 57f328136acc438bb0fefd739b67612e P 7cd2a1e3c7114fb68fe20199e8dd3782 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 622fa574e745488c9c18328a01285572 (127.0.98.67:45617), 26c6811dc2bc42fabfa7a3579aaec754 (127.0.98.65:33343)
W20260925 04:48:20.018457 18841 tablet.cc:2369] T 57f328136acc438bb0fefd739b67612e P 7cd2a1e3c7114fb68fe20199e8dd3782: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:48:20.022327 18989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:20.018912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:54063 (local address 127.0.98.65:33343)
0925 04:48:20.019142 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:20.019146 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:20.019158 (+ 12us) server_negotiation.cc:407] Connection header received
0925 04:48:20.019207 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:20.019210 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:20.019280 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:20.019356 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:48:20.019639 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:20.020152 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:20.021203 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:20.021619 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:20.021730 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:20.021799 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:20.022090 (+ 291us) server_negotiation.cc:299] Negotiation successful
0925 04:48:20.022166 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138}
I20260925 04:48:20.022383 18842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:20.018494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:45617 (local address 127.0.98.66:56939)
0925 04:48:20.018619 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:20.018634 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:48:20.018734 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:48:20.019286 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:48:20.019289 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:48:20.019307 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:48:20.019461 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:20.019468 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:20.020238 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:20.020241 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:20.021039 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:20.021047 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:20.021887 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:20.021890 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:20.022053 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:20.022077 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 04:48:20.022151 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 04:48:20.022219 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":27}
I20260925 04:48:20.022537 18990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:20.018856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:56939 (local address 127.0.98.67:45617)
0925 04:48:20.018980 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:20.018986 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:20.019001 (+ 15us) server_negotiation.cc:407] Connection header received
0925 04:48:20.019057 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:20.019060 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:20.019165 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:20.019248 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:48:20.019585 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:20.020112 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:20.021174 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:20.021747 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:20.021857 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:20.021927 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:20.022321 (+ 394us) server_negotiation.cc:299] Negotiation successful
0925 04:48:20.022413 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 04:48:20.022554 19000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:20.018541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:33343 (local address 127.0.98.66:54063)
0925 04:48:20.018862 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:20.018886 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 04:48:20.018987 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:48:20.019394 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:48:20.019396 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:48:20.019421 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:48:20.019512 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:20.019516 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:20.020268 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:20.020273 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:20.021073 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:20.021081 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:20.021767 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:20.021773 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:20.021890 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:20.021977 (+ 87us) client_negotiation.cc:769] Sending connection context
0925 04:48:20.022070 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 04:48:20.022149 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":34,"thread_start_us":90,"threads_started":1}
I20260925 04:48:20.023197 18660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57f328136acc438bb0fefd739b67612e" candidate_uuid: "7cd2a1e3c7114fb68fe20199e8dd3782" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c6811dc2bc42fabfa7a3579aaec754" is_pre_election: true
I20260925 04:48:20.023197 18928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57f328136acc438bb0fefd739b67612e" candidate_uuid: "7cd2a1e3c7114fb68fe20199e8dd3782" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "622fa574e745488c9c18328a01285572" is_pre_election: true
I20260925 04:48:20.023525 18928 raft_consensus.cc:2454] T 57f328136acc438bb0fefd739b67612e P 622fa574e745488c9c18328a01285572 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7cd2a1e3c7114fb68fe20199e8dd3782 in term 0.
I20260925 04:48:20.023525 18660 raft_consensus.cc:2454] T 57f328136acc438bb0fefd739b67612e P 26c6811dc2bc42fabfa7a3579aaec754 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7cd2a1e3c7114fb68fe20199e8dd3782 in term 0.
I20260925 04:48:20.024048 18730 leader_election.cc:302] T 57f328136acc438bb0fefd739b67612e P 7cd2a1e3c7114fb68fe20199e8dd3782 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 622fa574e745488c9c18328a01285572, 7cd2a1e3c7114fb68fe20199e8dd3782; no voters:
I20260925 04:48:20.024335 18998 raft_consensus.cc:2781] T 57f328136acc438bb0fefd739b67612e P 7cd2a1e3c7114fb68fe20199e8dd3782 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:48:20.024416 18998 raft_consensus.cc:482] T 57f328136acc438bb0fefd739b67612e P 7cd2a1e3c7114fb68fe20199e8dd3782 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:48:20.024499 18998 raft_consensus.cc:3037] T 57f328136acc438bb0fefd739b67612e P 7cd2a1e3c7114fb68fe20199e8dd3782 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:48:20.025518 18998 raft_consensus.cc:504] T 57f328136acc438bb0fefd739b67612e P 7cd2a1e3c7114fb68fe20199e8dd3782 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cd2a1e3c7114fb68fe20199e8dd3782" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37005 } } peers { permanent_uuid: "622fa574e745488c9c18328a01285572" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 45617 } } peers { permanent_uuid: "26c6811dc2bc42fabfa7a3579aaec754" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33343 } }
I20260925 04:48:20.025898 18998 leader_election.cc:288] T 57f328136acc438bb0fefd739b67612e P 7cd2a1e3c7114fb68fe20199e8dd3782 [CANDIDATE]: Term 1 election: Requested vote from peers 622fa574e745488c9c18328a01285572 (127.0.98.67:45617), 26c6811dc2bc42fabfa7a3579aaec754 (127.0.98.65:33343)
I20260925 04:48:20.026206 18928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57f328136acc438bb0fefd739b67612e" candidate_uuid: "7cd2a1e3c7114fb68fe20199e8dd3782" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "622fa574e745488c9c18328a01285572"
I20260925 04:48:20.026281 18660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57f328136acc438bb0fefd739b67612e" candidate_uuid: "7cd2a1e3c7114fb68fe20199e8dd3782" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c6811dc2bc42fabfa7a3579aaec754"
I20260925 04:48:20.026399 18928 raft_consensus.cc:3037] T 57f328136acc438bb0fefd739b67612e P 622fa574e745488c9c18328a01285572 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:48:20.026399 18660 raft_consensus.cc:3037] T 57f328136acc438bb0fefd739b67612e P 26c6811dc2bc42fabfa7a3579aaec754 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:48:20.027647 18660 raft_consensus.cc:2454] T 57f328136acc438bb0fefd739b67612e P 26c6811dc2bc42fabfa7a3579aaec754 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7cd2a1e3c7114fb68fe20199e8dd3782 in term 1.
I20260925 04:48:20.027647 18928 raft_consensus.cc:2454] T 57f328136acc438bb0fefd739b67612e P 622fa574e745488c9c18328a01285572 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7cd2a1e3c7114fb68fe20199e8dd3782 in term 1.
I20260925 04:48:20.027937 18727 leader_election.cc:302] T 57f328136acc438bb0fefd739b67612e P 7cd2a1e3c7114fb68fe20199e8dd3782 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26c6811dc2bc42fabfa7a3579aaec754, 7cd2a1e3c7114fb68fe20199e8dd3782; no voters:
I20260925 04:48:20.028110 18998 raft_consensus.cc:2781] T 57f328136acc438bb0fefd739b67612e P 7cd2a1e3c7114fb68fe20199e8dd3782 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:48:20.028384 18998 raft_consensus.cc:686] T 57f328136acc438bb0fefd739b67612e P 7cd2a1e3c7114fb68fe20199e8dd3782 [term 1 LEADER]: Becoming Leader. State: Replica: 7cd2a1e3c7114fb68fe20199e8dd3782, State: Running, Role: LEADER
I20260925 04:48:20.028728 18998 consensus_queue.cc:237] T 57f328136acc438bb0fefd739b67612e P 7cd2a1e3c7114fb68fe20199e8dd3782 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cd2a1e3c7114fb68fe20199e8dd3782" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37005 } } peers { permanent_uuid: "622fa574e745488c9c18328a01285572" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 45617 } } peers { permanent_uuid: "26c6811dc2bc42fabfa7a3579aaec754" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33343 } }
I20260925 04:48:20.031872 18513 catalog_manager.cc:5495] T 57f328136acc438bb0fefd739b67612e P 7cd2a1e3c7114fb68fe20199e8dd3782 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7cd2a1e3c7114fb68fe20199e8dd3782 (127.0.98.66). New cstate: current_term: 1 leader_uuid: "7cd2a1e3c7114fb68fe20199e8dd3782" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cd2a1e3c7114fb68fe20199e8dd3782" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "622fa574e745488c9c18328a01285572" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 45617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26c6811dc2bc42fabfa7a3579aaec754" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33343 } health_report { overall_health: UNKNOWN } } }
W20260925 04:48:20.060165 18707 tablet.cc:2369] T 57f328136acc438bb0fefd739b67612e P 26c6811dc2bc42fabfa7a3579aaec754: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:48:20.088675 18987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:20.084932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55196 (local address 127.0.98.66:37005)
0925 04:48:20.085096 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:20.085101 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:20.085116 (+ 15us) server_negotiation.cc:407] Connection header received
0925 04:48:20.085262 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:20.085265 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:20.085319 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:20.085443 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:48:20.085712 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:20.086230 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:20.087149 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:20.087370 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:20.087483 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:20.087789 (+ 306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:48:20.088202 (+ 413us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:48:20.088366 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:20.088465 (+ 99us) server_negotiation.cc:299] Negotiation successful
0925 04:48:20.088503 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 04:48:20.094766 18660 raft_consensus.cc:1261] T 57f328136acc438bb0fefd739b67612e P 26c6811dc2bc42fabfa7a3579aaec754 [term 1 FOLLOWER]: Refusing update from remote peer 7cd2a1e3c7114fb68fe20199e8dd3782: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:48:20.094763 18928 raft_consensus.cc:1261] T 57f328136acc438bb0fefd739b67612e P 622fa574e745488c9c18328a01285572 [term 1 FOLLOWER]: Refusing update from remote peer 7cd2a1e3c7114fb68fe20199e8dd3782: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:48:20.095501 18998 consensus_queue.cc:1035] T 57f328136acc438bb0fefd739b67612e P 7cd2a1e3c7114fb68fe20199e8dd3782 [LEADER]: Connected to new peer: Peer: permanent_uuid: "622fa574e745488c9c18328a01285572" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 45617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:48:20.095752 19001 consensus_queue.cc:1035] T 57f328136acc438bb0fefd739b67612e P 7cd2a1e3c7114fb68fe20199e8dd3782 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c6811dc2bc42fabfa7a3579aaec754" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:48:20.114038 18514 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34526:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:48:20.114229 18514 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34526:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:48:20.116793 18514 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 91d38bf70a3b4df69fb6d49baef5ad53: Sending DeleteTablet for 3 replicas of tablet 57f328136acc438bb0fefd739b67612e
I20260925 04:48:20.117631 18774 tablet_service.cc:1514] Processing DeleteTablet for tablet 57f328136acc438bb0fefd739b67612e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:48:20 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:55186
I20260925 04:48:20.117811 18631 tablet_service.cc:1514] Processing DeleteTablet for tablet 57f328136acc438bb0fefd739b67612e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:48:20 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:60672
I20260925 04:48:20.118238 19014 tablet_replica.cc:330] T 57f328136acc438bb0fefd739b67612e P 7cd2a1e3c7114fb68fe20199e8dd3782: stopping tablet replica
I20260925 04:48:20.118379 18908 tablet_service.cc:1514] Processing DeleteTablet for tablet 57f328136acc438bb0fefd739b67612e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:48:20 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:42152
I20260925 04:48:20.118593 19014 raft_consensus.cc:2229] T 57f328136acc438bb0fefd739b67612e P 7cd2a1e3c7114fb68fe20199e8dd3782 [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:48:20.118742 19015 tablet_replica.cc:330] T 57f328136acc438bb0fefd739b67612e P 26c6811dc2bc42fabfa7a3579aaec754: stopping tablet replica
I20260925 04:48:20.118824 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 18577
I20260925 04:48:20.120174 19016 tablet_replica.cc:330] T 57f328136acc438bb0fefd739b67612e P 622fa574e745488c9c18328a01285572: stopping tablet replica
I20260925 04:48:20.120519 19016 raft_consensus.cc:2229] T 57f328136acc438bb0fefd739b67612e P 622fa574e745488c9c18328a01285572 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:48:20.120853 19016 raft_consensus.cc:2258] T 57f328136acc438bb0fefd739b67612e P 622fa574e745488c9c18328a01285572 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:48:20.122663 19016 ts_tablet_manager.cc:1905] T 57f328136acc438bb0fefd739b67612e P 622fa574e745488c9c18328a01285572: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 04:48:20.125418 18499 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.65:33343 (error 108)
I20260925 04:48:20.125682 19016 ts_tablet_manager.cc:1918] T 57f328136acc438bb0fefd739b67612e P 622fa574e745488c9c18328a01285572: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:48:20.125897 19016 log.cc:1196] T 57f328136acc438bb0fefd739b67612e P 622fa574e745488c9c18328a01285572: Deleting WAL directory at /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790311449873301-393-0/minicluster-data/ts-2/wal/wals/57f328136acc438bb0fefd739b67612e
I20260925 04:48:20.126309 19016 ts_tablet_manager.cc:1939] T 57f328136acc438bb0fefd739b67612e P 622fa574e745488c9c18328a01285572: Deleting consensus metadata
I20260925 04:48:20.126533 18988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:20.126174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:33343 (local address 127.0.0.1:60684)
0925 04:48:20.126343 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:20.126419 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.65:33343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260925 04:48:20.126613 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 18711
I20260925 04:48:20.126616 19014 raft_consensus.cc:2258] T 57f328136acc438bb0fefd739b67612e P 7cd2a1e3c7114fb68fe20199e8dd3782 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260925 04:48:20.126847 18499 catalog_manager.cc:4584] TS 26c6811dc2bc42fabfa7a3579aaec754 (127.0.98.65:33343): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 57f328136acc438bb0fefd739b67612e: Network error: Client connection negotiation failed: client connection to 127.0.98.65:33343: connect: Connection refused (error 111)
I20260925 04:48:20.127720 18502 catalog_manager.cc:4841] TS 622fa574e745488c9c18328a01285572 (127.0.98.67:45617): tablet 57f328136acc438bb0fefd739b67612e (table test-table [id=16d2e57b736d43d88317fab2996d59d7]) successfully deleted
I20260925 04:48:20.133816 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 18845
I20260925 04:48:20.134757 18988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:20.134065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:37005 (local address 127.0.0.1:55208)
0925 04:48:20.134585 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:20.134652 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.66:37005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415}
I20260925 04:48:20.139359 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 18483
2026-09-25T04:48:20Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2476 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-25T04:48:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:48:20Z Disabled control of system clock
I20260925 04:48:20.169809 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311449873301-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311449873301-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311449873301-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:36323
--webserver_interface=127.0.98.126
--webserver_port=0
--builtin_ntp_servers=127.0.98.84:38913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:36323
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 04:48:20.280529 19021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:48:20.280809 19021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:48:20.280848 19021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:48:20.284883 19021 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:48:20.284938 19021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:48:20.284955 19021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:48:20.284972 19021 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:48:20.284988 19021 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:48:20.289675 19021 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:38913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311449873301-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311449873301-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:36323
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:36323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311449873301-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:48:20.290732 19021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:48:20.292670 19021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:48:20.301507 19026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:48:20.301496 19029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:48:20.301707 19021 server_base.cc:1034] running on GCE node
W20260925 04:48:20.301496 19027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:48:20.302467 19021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:48:20.303679 19021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:48:20.304869 19021 hybrid_clock.cc:648] HybridClock initialized: now 1790311700304847 us; error 43 us; skew 500 ppm
I20260925 04:48:20.306972 19021 webserver.cc:460] Webserver started at http://127.0.98.126:34197/ using document root <none> and password file <none>
I20260925 04:48:20.307585 19021 fs_manager.cc:359] Metadata directory not provided
I20260925 04:48:20.307680 19021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:48:20.307924 19021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:48:20.310020 19021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311449873301-393-0/minicluster-data/master-0/data/instance:
uuid: "04540d34205f41149b437abb149f3f8c"
format_stamp: "Formatted at 2026-09-25 04:48:20 on dist-test-slave-kghz"
server_key: "ce0d8e9e498ce7216b5b228f8c9ee0db"
server_key_iv: "592e6069cfc441335483a96a874e7993"
server_key_version: "encryptionkey@0"
I20260925 04:48:20.310657 19021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311449873301-393-0/minicluster-data/master-0/wal/instance:
uuid: "04540d34205f41149b437abb149f3f8c"
format_stamp: "Formatted at 2026-09-25 04:48:20 on dist-test-slave-kghz"
server_key: "ce0d8e9e498ce7216b5b228f8c9ee0db"
server_key_iv: "592e6069cfc441335483a96a874e7993"
server_key_version: "encryptionkey@0"
I20260925 04:48:20.314563 19021 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 04:48:20.316996 19035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:48:20.318315 19021 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 04:48:20.318459 19021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311449873301-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311449873301-393-0/minicluster-data/master-0/wal
uuid: "04540d34205f41149b437abb149f3f8c"
format_stamp: "Formatted at 2026-09-25 04:48:20 on dist-test-slave-kghz"
server_key: "ce0d8e9e498ce7216b5b228f8c9ee0db"
server_key_iv: "592e6069cfc441335483a96a874e7993"
server_key_version: "encryptionkey@0"
I20260925 04:48:20.318575 19021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311449873301-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311449873301-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311449873301-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:48:20.342089 19021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:48:20.342873 19021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:48:20.343101 19021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:48:20.350345 19021 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:36323
I20260925 04:48:20.350375 19087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:36323 every 8 connection(s)
I20260925 04:48:20.351343 19021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
I20260925 04:48:20.354643 19088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:48:20.355703 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 19021
I20260925 04:48:20.355829 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311449873301-393-0/minicluster-data/master-0/wal/instance
I20260925 04:48:20.356175 393 external_mini_cluster.cc:1428] Setting key e427a4b463a6cd0b417108a5a6b4caf1
I20260925 04:48:20.360988 19088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04540d34205f41149b437abb149f3f8c: Bootstrap starting.
I20260925 04:48:20.363663 19088 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04540d34205f41149b437abb149f3f8c: Neither blocks nor log segments found. Creating new log.
I20260925 04:48:20.364578 19088 log.cc:824] T 00000000000000000000000000000000 P 04540d34205f41149b437abb149f3f8c: Log is configured to *not* fsync() on all Append() calls
I20260925 04:48:20.366186 19091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:20.357757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44822 (local address 127.0.98.126:36323)
0925 04:48:20.358489 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:20.358505 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:20.358543 (+ 38us) server_negotiation.cc:407] Connection header received
0925 04:48:20.359212 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:20.359229 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:20.359634 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:20.359980 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:20.360434 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:20.361473 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:20.362430 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:20.363127 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:20.363313 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:20.364237 (+ 924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:20.364242 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:20.364256 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:48:20.364660 (+ 404us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:20.364803 (+ 143us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:48:20.364830 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:20.364914 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:20.365106 (+ 192us) server_negotiation.cc:299] Negotiation successful
0925 04:48:20.365404 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":123,"threads_started":1}
I20260925 04:48:20.366969 19088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04540d34205f41149b437abb149f3f8c: No bootstrap required, opened a new log
I20260925 04:48:20.370741 19088 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04540d34205f41149b437abb149f3f8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04540d34205f41149b437abb149f3f8c" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 36323 } }
I20260925 04:48:20.370972 19088 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04540d34205f41149b437abb149f3f8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:48:20.371006 19088 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04540d34205f41149b437abb149f3f8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04540d34205f41149b437abb149f3f8c, State: Initialized, Role: FOLLOWER
I20260925 04:48:20.371418 19088 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04540d34205f41149b437abb149f3f8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04540d34205f41149b437abb149f3f8c" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 36323 } }
I20260925 04:48:20.371552 19088 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04540d34205f41149b437abb149f3f8c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:48:20.371599 19088 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04540d34205f41149b437abb149f3f8c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:48:20.371672 19088 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04540d34205f41149b437abb149f3f8c [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:48:20.372576 19088 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04540d34205f41149b437abb149f3f8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04540d34205f41149b437abb149f3f8c" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 36323 } }
I20260925 04:48:20.372880 19088 leader_election.cc:302] T 00000000000000000000000000000000 P 04540d34205f41149b437abb149f3f8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04540d34205f41149b437abb149f3f8c; no voters:
I20260925 04:48:20.373145 19088 leader_election.cc:288] T 00000000000000000000000000000000 P 04540d34205f41149b437abb149f3f8c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:48:20.373287 19093 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04540d34205f41149b437abb149f3f8c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:48:20.373551 19093 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04540d34205f41149b437abb149f3f8c [term 1 LEADER]: Becoming Leader. State: Replica: 04540d34205f41149b437abb149f3f8c, State: Running, Role: LEADER
I20260925 04:48:20.373865 19093 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04540d34205f41149b437abb149f3f8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04540d34205f41149b437abb149f3f8c" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 36323 } }
I20260925 04:48:20.374529 19088 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04540d34205f41149b437abb149f3f8c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:48:20.375784 19094 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04540d34205f41149b437abb149f3f8c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04540d34205f41149b437abb149f3f8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04540d34205f41149b437abb149f3f8c" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 36323 } } }
I20260925 04:48:20.375932 19094 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04540d34205f41149b437abb149f3f8c [sys.catalog]: This master's current role is: LEADER
I20260925 04:48:20.376209 19095 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04540d34205f41149b437abb149f3f8c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04540d34205f41149b437abb149f3f8c. Latest consensus state: current_term: 1 leader_uuid: "04540d34205f41149b437abb149f3f8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04540d34205f41149b437abb149f3f8c" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 36323 } } }
I20260925 04:48:20.376318 19095 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04540d34205f41149b437abb149f3f8c [sys.catalog]: This master's current role is: LEADER
I20260925 04:48:20.376374 19102 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:48:20.379568 19102 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:48:20.385644 19102 catalog_manager.cc:1347] Generated new cluster ID: 15e3b57ad83346b7bed838b64945d118
I20260925 04:48:20.385735 19102 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:48:20.392711 19102 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:48:20.393646 19102 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:48:20.402144 19102 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04540d34205f41149b437abb149f3f8c: Generated new TSK 0
I20260925 04:48:20.403097 19102 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:48:20.412495 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311449873301-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311449873301-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311449873301-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:0
--local_ip_for_outbound_sockets=127.0.98.65
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:36323
--builtin_ntp_servers=127.0.98.84:38913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 04:48:20.531158 19112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:48:20.531415 19112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:48:20.531454 19112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:48:20.535362 19112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:48:20.535410 19112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:48:20.535463 19112 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260925 04:48:20.541627 19112 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:38913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311449873301-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311449873301-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:36323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19112
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311449873301-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:48:20.542714 19112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:48:20.544054 19112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:48:20.553568 19118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:48:20.553596 19120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:48:20.553610 19112 server_base.cc:1034] running on GCE node
W20260925 04:48:20.553632 19117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:48:20.554411 19112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:48:20.555050 19112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:48:20.556227 19112 hybrid_clock.cc:648] HybridClock initialized: now 1790311700556192 us; error 46 us; skew 500 ppm
I20260925 04:48:20.558149 19112 webserver.cc:460] Webserver started at http://127.0.98.65:41501/ using document root <none> and password file <none>
I20260925 04:48:20.558813 19112 fs_manager.cc:359] Metadata directory not provided
I20260925 04:48:20.558881 19112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:48:20.559113 19112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:48:20.561319 19112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311449873301-393-0/minicluster-data/ts-0/data/instance:
uuid: "46c3b1dd180748369dd5fb5be8fda535"
format_stamp: "Formatted at 2026-09-25 04:48:20 on dist-test-slave-kghz"
server_key: "e804ae7abc586a3be116ef5243672f49"
server_key_iv: "68c6552cf29b288a9ba489d8641e35df"
server_key_version: "encryptionkey@0"
I20260925 04:48:20.561926 19112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311449873301-393-0/minicluster-data/ts-0/wal/instance:
uuid: "46c3b1dd180748369dd5fb5be8fda535"
format_stamp: "Formatted at 2026-09-25 04:48:20 on dist-test-slave-kghz"
server_key: "e804ae7abc586a3be116ef5243672f49"
server_key_iv: "68c6552cf29b288a9ba489d8641e35df"
server_key_version: "encryptionkey@0"
I20260925 04:48:20.565811 19112 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 04:48:20.568392 19126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:48:20.569729 19112 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 04:48:20.569891 19112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311449873301-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311449873301-393-0/minicluster-data/ts-0/wal
uuid: "46c3b1dd180748369dd5fb5be8fda535"
format_stamp: "Formatted at 2026-09-25 04:48:20 on dist-test-slave-kghz"
server_key: "e804ae7abc586a3be116ef5243672f49"
server_key_iv: "68c6552cf29b288a9ba489d8641e35df"
server_key_version: "encryptionkey@0"
I20260925 04:48:20.570014 19112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311449873301-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311449873301-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311449873301-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:48:20.596063 19112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:48:20.596793 19112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:48:20.596997 19112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:48:20.597611 19112 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:48:20.598591 19112 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:48:20.598662 19112 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:48:20.598731 19112 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:48:20.598774 19112 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:48:20.607535 19112 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:43681
I20260925 04:48:20.607568 19239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:43681 every 8 connection(s)
I20260925 04:48:20.608433 19112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
I20260925 04:48:20.609725 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 19112
I20260925 04:48:20.609880 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311449873301-393-0/minicluster-data/ts-0/wal/instance
I20260925 04:48:20.610200 393 external_mini_cluster.cc:1428] Setting key c22e845096724011cb3cc578694d0563
I20260925 04:48:20.612314 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311449873301-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311449873301-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311449873301-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:0
--local_ip_for_outbound_sockets=127.0.98.66
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:36323
--builtin_ntp_servers=127.0.98.84:38913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 04:48:20.618436 19091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:20.610434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:44523 (local address 127.0.98.126:36323)
0925 04:48:20.610608 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:20.610616 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:20.611911 (+ 1295us) server_negotiation.cc:407] Connection header received
0925 04:48:20.613424 (+ 1513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:20.613428 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:20.613481 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:20.613565 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:20.614624 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:20.615174 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:20.616326 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:20.616597 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:20.616726 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:20.617898 (+ 1172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:20.617906 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:20.617908 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:48:20.617920 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:20.617937 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:48:20.617945 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:20.618023 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:20.618197 (+ 174us) server_negotiation.cc:299] Negotiation successful
0925 04:48:20.618291 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 04:48:20.619304 19242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:20.610957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:36323 (local address 127.0.98.65:44523)
0925 04:48:20.611733 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:20.611792 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 04:48:20.613193 (+ 1401us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:48:20.613731 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:48:20.613738 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:48:20.613822 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:48:20.614464 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:20.614481 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:20.615308 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:20.615311 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:20.616187 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:20.616196 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:20.616757 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:20.616761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:20.616931 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:20.617619 (+ 688us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:48:20.617621 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:48:20.617672 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:48:20.617738 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:48:20.618037 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:48:20.618049 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:48:20.618056 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 04:48:20.618196 (+ 140us) client_negotiation.cc:240] Negotiation successful
0925 04:48:20.618506 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":112,"threads_started":1}
I20260925 04:48:20.620616 19240 heartbeater.cc:347] Connected to a master server at 127.0.98.126:36323
I20260925 04:48:20.620873 19240 heartbeater.cc:464] Registering TS with master...
I20260925 04:48:20.621412 19240 heartbeater.cc:511] Master 127.0.98.126:36323 requested a full tablet report, sending...
I20260925 04:48:20.623171 19051 ts_manager.cc:194] Registered new tserver with Master: 46c3b1dd180748369dd5fb5be8fda535 (127.0.98.65:43681)
I20260925 04:48:20.624325 19051 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.65:44523
W20260925 04:48:20.726882 19243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:48:20.727172 19243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:48:20.727243 19243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:48:20.731256 19243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:48:20.731333 19243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:48:20.731423 19243 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260925 04:48:20.736016 19243 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:38913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311449873301-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311449873301-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:36323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19243
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311449873301-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:48:20.737085 19243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:48:20.738624 19243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:48:20.748626 19248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:48:20.748626 19251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:48:20.748683 19249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:48:20.748939 19243 server_base.cc:1034] running on GCE node
I20260925 04:48:20.749477 19243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:48:20.750154 19243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:48:20.751332 19243 hybrid_clock.cc:648] HybridClock initialized: now 1790311700751310 us; error 30 us; skew 500 ppm
I20260925 04:48:20.753413 19243 webserver.cc:460] Webserver started at http://127.0.98.66:43717/ using document root <none> and password file <none>
I20260925 04:48:20.754031 19243 fs_manager.cc:359] Metadata directory not provided
I20260925 04:48:20.754098 19243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:48:20.754390 19243 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:48:20.756567 19243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311449873301-393-0/minicluster-data/ts-1/data/instance:
uuid: "7d0056b98dac41e1b5044e476d03feaa"
format_stamp: "Formatted at 2026-09-25 04:48:20 on dist-test-slave-kghz"
server_key: "fd7ccb8e4e46d1252ac4833b84a99c8d"
server_key_iv: "0abcbd4f9f7bafe470f185f73e314f5c"
server_key_version: "encryptionkey@0"
I20260925 04:48:20.757162 19243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311449873301-393-0/minicluster-data/ts-1/wal/instance:
uuid: "7d0056b98dac41e1b5044e476d03feaa"
format_stamp: "Formatted at 2026-09-25 04:48:20 on dist-test-slave-kghz"
server_key: "fd7ccb8e4e46d1252ac4833b84a99c8d"
server_key_iv: "0abcbd4f9f7bafe470f185f73e314f5c"
server_key_version: "encryptionkey@0"
I20260925 04:48:20.761092 19243 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260925 04:48:20.763708 19257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:48:20.765030 19243 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 04:48:20.765161 19243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311449873301-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311449873301-393-0/minicluster-data/ts-1/wal
uuid: "7d0056b98dac41e1b5044e476d03feaa"
format_stamp: "Formatted at 2026-09-25 04:48:20 on dist-test-slave-kghz"
server_key: "fd7ccb8e4e46d1252ac4833b84a99c8d"
server_key_iv: "0abcbd4f9f7bafe470f185f73e314f5c"
server_key_version: "encryptionkey@0"
I20260925 04:48:20.765292 19243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311449873301-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311449873301-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311449873301-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:48:20.781392 19243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:48:20.782116 19243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:48:20.782373 19243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:48:20.783001 19243 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:48:20.784008 19243 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:48:20.784057 19243 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:48:20.784099 19243 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:48:20.784161 19243 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:48:20.794008 19243 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:42717
I20260925 04:48:20.794032 19370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:42717 every 8 connection(s)
I20260925 04:48:20.795270 19243 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
I20260925 04:48:20.799021 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 19243
I20260925 04:48:20.799144 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311449873301-393-0/minicluster-data/ts-1/wal/instance
I20260925 04:48:20.799456 393 external_mini_cluster.cc:1428] Setting key d756e1a4646cfb0f00eea911ae83b6a7
I20260925 04:48:20.801744 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311449873301-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311449873301-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311449873301-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:0
--local_ip_for_outbound_sockets=127.0.98.67
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:36323
--builtin_ntp_servers=127.0.98.84:38913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 04:48:20.806139 19091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:20.797181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:38405 (local address 127.0.98.126:36323)
0925 04:48:20.797347 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:20.797352 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:20.798519 (+ 1167us) server_negotiation.cc:407] Connection header received
0925 04:48:20.800328 (+ 1809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:20.800331 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:20.800379 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:20.800451 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:20.801617 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:20.802232 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:20.803809 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:20.804175 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:20.804311 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:20.805509 (+ 1198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:20.805516 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:20.805522 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:48:20.805540 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:20.805568 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:48:20.805581 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:20.805670 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:20.805847 (+ 177us) server_negotiation.cc:299] Negotiation successful
0925 04:48:20.805928 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 04:48:20.806926 19373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:20.797628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:36323 (local address 127.0.98.66:38405)
0925 04:48:20.798365 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:20.798416 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 04:48:20.800101 (+ 1685us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:48:20.800638 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:48:20.800647 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:48:20.800806 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:48:20.801442 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:20.801462 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:20.802408 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:20.802415 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:20.803640 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:20.803650 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:20.804326 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:20.804329 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:20.804471 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:20.805185 (+ 714us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:48:20.805187 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:48:20.805229 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:48:20.805322 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:48:20.805674 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:48:20.805682 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:48:20.805686 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 04:48:20.805810 (+ 124us) client_negotiation.cc:240] Negotiation successful
0925 04:48:20.806036 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":110,"threads_started":1}
I20260925 04:48:20.808172 19371 heartbeater.cc:347] Connected to a master server at 127.0.98.126:36323
I20260925 04:48:20.808413 19371 heartbeater.cc:464] Registering TS with master...
I20260925 04:48:20.808960 19371 heartbeater.cc:511] Master 127.0.98.126:36323 requested a full tablet report, sending...
I20260925 04:48:20.810048 19051 ts_manager.cc:194] Registered new tserver with Master: 7d0056b98dac41e1b5044e476d03feaa (127.0.98.66:42717)
I20260925 04:48:20.810729 19051 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.66:38405
W20260925 04:48:20.913676 19374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:48:20.913955 19374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:48:20.913995 19374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:48:20.918229 19374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:48:20.918350 19374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:48:20.918447 19374 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260925 04:48:20.923416 19374 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:38913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311449873301-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311449873301-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:36323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311449873301-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:48:20.924508 19374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:48:20.925925 19374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:48:20.935880 19379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:48:20.935880 19380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:48:20.935932 19374 server_base.cc:1034] running on GCE node
W20260925 04:48:20.935930 19382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:48:20.936573 19374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:48:20.937152 19374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:48:20.938391 19374 hybrid_clock.cc:648] HybridClock initialized: now 1790311700938365 us; error 33 us; skew 500 ppm
I20260925 04:48:20.940416 19374 webserver.cc:460] Webserver started at http://127.0.98.67:36417/ using document root <none> and password file <none>
I20260925 04:48:20.940985 19374 fs_manager.cc:359] Metadata directory not provided
I20260925 04:48:20.941047 19374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:48:20.941252 19374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:48:20.943423 19374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311449873301-393-0/minicluster-data/ts-2/data/instance:
uuid: "0f1e0204b7b14908b28c0c4d26d19b25"
format_stamp: "Formatted at 2026-09-25 04:48:20 on dist-test-slave-kghz"
server_key: "da56e86b43e3d86a59e0b0a38418af98"
server_key_iv: "e972c1ad81876bc1300b2dccaa6896c6"
server_key_version: "encryptionkey@0"
I20260925 04:48:20.943953 19374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311449873301-393-0/minicluster-data/ts-2/wal/instance:
uuid: "0f1e0204b7b14908b28c0c4d26d19b25"
format_stamp: "Formatted at 2026-09-25 04:48:20 on dist-test-slave-kghz"
server_key: "da56e86b43e3d86a59e0b0a38418af98"
server_key_iv: "e972c1ad81876bc1300b2dccaa6896c6"
server_key_version: "encryptionkey@0"
I20260925 04:48:20.947901 19374 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260925 04:48:20.950418 19388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:48:20.951719 19374 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 04:48:20.951830 19374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311449873301-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311449873301-393-0/minicluster-data/ts-2/wal
uuid: "0f1e0204b7b14908b28c0c4d26d19b25"
format_stamp: "Formatted at 2026-09-25 04:48:20 on dist-test-slave-kghz"
server_key: "da56e86b43e3d86a59e0b0a38418af98"
server_key_iv: "e972c1ad81876bc1300b2dccaa6896c6"
server_key_version: "encryptionkey@0"
I20260925 04:48:20.951924 19374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311449873301-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311449873301-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311449873301-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:48:20.979050 19374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:48:20.979799 19374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:48:20.979962 19374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:48:20.980521 19374 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:48:20.981560 19374 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:48:20.981639 19374 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:48:20.981721 19374 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:48:20.981767 19374 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:48:20.990978 19374 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:37297
I20260925 04:48:20.991012 19501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:37297 every 8 connection(s)
I20260925 04:48:20.991977 19374 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
I20260925 04:48:20.998628 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 19374
I20260925 04:48:20.998718 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311449873301-393-0/minicluster-data/ts-2/wal/instance
I20260925 04:48:20.998997 393 external_mini_cluster.cc:1428] Setting key f07cc24169c9f24073ca9a89ae3285b2
I20260925 04:48:21.001821 19091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:20.993926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:37921 (local address 127.0.98.126:36323)
0925 04:48:20.994073 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:20.994076 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:20.995242 (+ 1166us) server_negotiation.cc:407] Connection header received
0925 04:48:20.996660 (+ 1418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:20.996664 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:20.996734 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:20.996840 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:20.997765 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:20.998323 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:20.999714 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:21.000033 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:21.000145 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:21.001308 (+ 1163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:21.001312 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:21.001315 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:48:21.001327 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:21.001349 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:48:21.001364 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:21.001441 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:21.001609 (+ 168us) server_negotiation.cc:299] Negotiation successful
0925 04:48:21.001670 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 04:48:21.002767 19504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:20.994364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:36323 (local address 127.0.98.67:37921)
0925 04:48:20.995064 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:20.995117 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 04:48:20.996438 (+ 1321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:48:20.997020 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:48:20.997027 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:48:20.997113 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:48:20.997601 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:20.997613 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:20.998464 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:20.998466 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:20.999584 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:20.999593 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:21.000184 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:21.000188 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:21.000366 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:21.001030 (+ 664us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:48:21.001032 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:48:21.001089 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:48:21.001153 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:48:21.001444 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:48:21.001455 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:48:21.001462 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 04:48:21.001599 (+ 137us) client_negotiation.cc:240] Negotiation successful
0925 04:48:21.001884 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":119,"threads_started":1}
I20260925 04:48:21.003861 19502 heartbeater.cc:347] Connected to a master server at 127.0.98.126:36323
I20260925 04:48:21.004103 19502 heartbeater.cc:464] Registering TS with master...
I20260925 04:48:21.004644 19502 heartbeater.cc:511] Master 127.0.98.126:36323 requested a full tablet report, sending...
I20260925 04:48:21.005681 19051 ts_manager.cc:194] Registered new tserver with Master: 0f1e0204b7b14908b28c0c4d26d19b25 (127.0.98.67:37297)
I20260925 04:48:21.006357 19051 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.67:37921
I20260925 04:48:21.013015 393 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 04:48:21.019048 19091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:21.015000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44826 (local address 127.0.98.126:36323)
0925 04:48:21.015154 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:21.015158 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:21.015348 (+ 190us) server_negotiation.cc:407] Connection header received
0925 04:48:21.015513 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:21.015516 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:21.015568 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:21.015647 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:21.015929 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:21.016443 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:21.017296 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:21.017530 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:21.017640 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:21.018539 (+ 899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:21.018543 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:21.018545 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:48:21.018558 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:21.018586 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:48:21.018595 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:21.018726 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:21.018850 (+ 124us) server_negotiation.cc:299] Negotiation successful
0925 04:48:21.018888 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 04:48:21.022024 19051 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44826:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:48:21.025138 19051 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:48:21.039320 19515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:21.033332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37962 (local address 127.0.98.67:37297)
0925 04:48:21.034319 (+ 987us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:21.034330 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:21.034350 (+ 20us) server_negotiation.cc:407] Connection header received
0925 04:48:21.034413 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:21.034418 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:21.034567 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:21.034704 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:48:21.035130 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:21.036111 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:21.038000 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:21.038640 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:21.038770 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:21.038945 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:21.039053 (+ 108us) server_negotiation.cc:299] Negotiation successful
0925 04:48:21.039145 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":872,"thread_start_us":148,"threads_started":1}
I20260925 04:48:21.039348 19514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:21.033195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:37297 (local address 127.0.0.1:37962)
0925 04:48:21.033874 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:21.033924 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 04:48:21.034064 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:48:21.034876 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:48:21.034881 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:48:21.034911 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:48:21.035023 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:21.035031 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:21.036216 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:21.036221 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:21.037831 (+ 1610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:21.037838 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:21.038769 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:21.038773 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:21.038872 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:21.038908 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 04:48:21.039024 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 04:48:21.039142 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"mutex_wait_us":31,"spinlock_wait_cycles":5504,"thread_start_us":80,"threads_started":1}
I20260925 04:48:21.039394 19513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:21.033274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:42717 (local address 127.0.0.1:35544)
0925 04:48:21.033875 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:21.033924 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 04:48:21.034064 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:48:21.034828 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:48:21.034836 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:48:21.034859 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:48:21.035004 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:21.035015 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:21.036442 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:21.036445 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:21.037624 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:21.037633 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:21.038963 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:21.038969 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:21.039064 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:21.039086 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 04:48:21.039152 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 04:48:21.039221 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"mutex_wait_us":26,"spinlock_wait_cycles":10752,"thread_start_us":106,"threads_started":1}
I20260925 04:48:21.039767 19516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:21.033635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35544 (local address 127.0.98.66:42717)
0925 04:48:21.034049 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:21.034061 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:21.034088 (+ 27us) server_negotiation.cc:407] Connection header received
0925 04:48:21.034273 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:21.034279 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:21.034479 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:21.034633 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:48:21.035145 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:21.036325 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:21.037777 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:21.038597 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:21.038702 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:21.038891 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:21.039509 (+ 618us) server_negotiation.cc:299] Negotiation successful
0925 04:48:21.039605 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":124,"threads_started":1}
I20260925 04:48:21.039971 19518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:21.034300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56152 (local address 127.0.98.65:43681)
0925 04:48:21.034980 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:21.034991 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:21.035292 (+ 301us) server_negotiation.cc:407] Connection header received
0925 04:48:21.035370 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:21.035375 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:21.035546 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:21.035686 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:48:21.035908 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:21.036813 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:21.038149 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:21.038944 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:21.039044 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:21.039207 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:21.039701 (+ 494us) server_negotiation.cc:299] Negotiation successful
0925 04:48:21.039801 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":121,"threads_started":1}
I20260925 04:48:21.039610 19517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:21.033910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:43681 (local address 127.0.0.1:56152)
0925 04:48:21.035181 (+ 1271us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:21.035195 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:48:21.035280 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:48:21.035694 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:48:21.035697 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:48:21.035716 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:48:21.035790 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:21.035795 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:21.036922 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:21.036928 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:21.038019 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:21.038029 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:21.039266 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:21.039269 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:21.039350 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:21.039369 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 04:48:21.039439 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 04:48:21.039508 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722,"thread_start_us":88,"threads_started":1}
I20260925 04:48:21.042658 19305 tablet_service.cc:1467] Processing CreateTablet for tablet 036da4f3cef94499ae4b3d9b8ab82e8e (DEFAULT_TABLE table=test-table [id=0a56915787b64bd08b18adef2ecf9f77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:48:21.042662 19174 tablet_service.cc:1467] Processing CreateTablet for tablet 036da4f3cef94499ae4b3d9b8ab82e8e (DEFAULT_TABLE table=test-table [id=0a56915787b64bd08b18adef2ecf9f77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:48:21.043527 19305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 036da4f3cef94499ae4b3d9b8ab82e8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:48:21.043527 19174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 036da4f3cef94499ae4b3d9b8ab82e8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:48:21.046502 19436 tablet_service.cc:1467] Processing CreateTablet for tablet 036da4f3cef94499ae4b3d9b8ab82e8e (DEFAULT_TABLE table=test-table [id=0a56915787b64bd08b18adef2ecf9f77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:48:21.047569 19436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 036da4f3cef94499ae4b3d9b8ab82e8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:48:21.050432 19520 tablet_bootstrap.cc:492] T 036da4f3cef94499ae4b3d9b8ab82e8e P 7d0056b98dac41e1b5044e476d03feaa: Bootstrap starting.
I20260925 04:48:21.051327 19519 tablet_bootstrap.cc:492] T 036da4f3cef94499ae4b3d9b8ab82e8e P 46c3b1dd180748369dd5fb5be8fda535: Bootstrap starting.
I20260925 04:48:21.052515 19520 tablet_bootstrap.cc:654] T 036da4f3cef94499ae4b3d9b8ab82e8e P 7d0056b98dac41e1b5044e476d03feaa: Neither blocks nor log segments found. Creating new log.
I20260925 04:48:21.053254 19520 log.cc:824] T 036da4f3cef94499ae4b3d9b8ab82e8e P 7d0056b98dac41e1b5044e476d03feaa: Log is configured to *not* fsync() on all Append() calls
I20260925 04:48:21.053490 19519 tablet_bootstrap.cc:654] T 036da4f3cef94499ae4b3d9b8ab82e8e P 46c3b1dd180748369dd5fb5be8fda535: Neither blocks nor log segments found. Creating new log.
I20260925 04:48:21.054328 19519 log.cc:824] T 036da4f3cef94499ae4b3d9b8ab82e8e P 46c3b1dd180748369dd5fb5be8fda535: Log is configured to *not* fsync() on all Append() calls
I20260925 04:48:21.054708 19522 tablet_bootstrap.cc:492] T 036da4f3cef94499ae4b3d9b8ab82e8e P 0f1e0204b7b14908b28c0c4d26d19b25: Bootstrap starting.
I20260925 04:48:21.055442 19520 tablet_bootstrap.cc:492] T 036da4f3cef94499ae4b3d9b8ab82e8e P 7d0056b98dac41e1b5044e476d03feaa: No bootstrap required, opened a new log
I20260925 04:48:21.055647 19520 ts_tablet_manager.cc:1397] T 036da4f3cef94499ae4b3d9b8ab82e8e P 7d0056b98dac41e1b5044e476d03feaa: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 04:48:21.056286 19519 tablet_bootstrap.cc:492] T 036da4f3cef94499ae4b3d9b8ab82e8e P 46c3b1dd180748369dd5fb5be8fda535: No bootstrap required, opened a new log
I20260925 04:48:21.056488 19519 ts_tablet_manager.cc:1397] T 036da4f3cef94499ae4b3d9b8ab82e8e P 46c3b1dd180748369dd5fb5be8fda535: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260925 04:48:21.057463 19522 tablet_bootstrap.cc:654] T 036da4f3cef94499ae4b3d9b8ab82e8e P 0f1e0204b7b14908b28c0c4d26d19b25: Neither blocks nor log segments found. Creating new log.
I20260925 04:48:21.058337 19522 log.cc:824] T 036da4f3cef94499ae4b3d9b8ab82e8e P 0f1e0204b7b14908b28c0c4d26d19b25: Log is configured to *not* fsync() on all Append() calls
I20260925 04:48:21.060012 19519 raft_consensus.cc:348] T 036da4f3cef94499ae4b3d9b8ab82e8e P 46c3b1dd180748369dd5fb5be8fda535 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f1e0204b7b14908b28c0c4d26d19b25" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 37297 } } peers { permanent_uuid: "7d0056b98dac41e1b5044e476d03feaa" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 42717 } } peers { permanent_uuid: "46c3b1dd180748369dd5fb5be8fda535" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43681 } }
I20260925 04:48:21.060056 19520 raft_consensus.cc:348] T 036da4f3cef94499ae4b3d9b8ab82e8e P 7d0056b98dac41e1b5044e476d03feaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f1e0204b7b14908b28c0c4d26d19b25" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 37297 } } peers { permanent_uuid: "7d0056b98dac41e1b5044e476d03feaa" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 42717 } } peers { permanent_uuid: "46c3b1dd180748369dd5fb5be8fda535" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43681 } }
I20260925 04:48:21.060250 19519 raft_consensus.cc:374] T 036da4f3cef94499ae4b3d9b8ab82e8e P 46c3b1dd180748369dd5fb5be8fda535 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:48:21.060298 19519 raft_consensus.cc:729] T 036da4f3cef94499ae4b3d9b8ab82e8e P 46c3b1dd180748369dd5fb5be8fda535 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46c3b1dd180748369dd5fb5be8fda535, State: Initialized, Role: FOLLOWER
I20260925 04:48:21.060307 19520 raft_consensus.cc:374] T 036da4f3cef94499ae4b3d9b8ab82e8e P 7d0056b98dac41e1b5044e476d03feaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:48:21.060328 19522 tablet_bootstrap.cc:492] T 036da4f3cef94499ae4b3d9b8ab82e8e P 0f1e0204b7b14908b28c0c4d26d19b25: No bootstrap required, opened a new log
I20260925 04:48:21.060361 19520 raft_consensus.cc:729] T 036da4f3cef94499ae4b3d9b8ab82e8e P 7d0056b98dac41e1b5044e476d03feaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d0056b98dac41e1b5044e476d03feaa, State: Initialized, Role: FOLLOWER
I20260925 04:48:21.060496 19522 ts_tablet_manager.cc:1397] T 036da4f3cef94499ae4b3d9b8ab82e8e P 0f1e0204b7b14908b28c0c4d26d19b25: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260925 04:48:21.060729 19519 consensus_queue.cc:260] T 036da4f3cef94499ae4b3d9b8ab82e8e P 46c3b1dd180748369dd5fb5be8fda535 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f1e0204b7b14908b28c0c4d26d19b25" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 37297 } } peers { permanent_uuid: "7d0056b98dac41e1b5044e476d03feaa" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 42717 } } peers { permanent_uuid: "46c3b1dd180748369dd5fb5be8fda535" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43681 } }
I20260925 04:48:21.060853 19520 consensus_queue.cc:260] T 036da4f3cef94499ae4b3d9b8ab82e8e P 7d0056b98dac41e1b5044e476d03feaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f1e0204b7b14908b28c0c4d26d19b25" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 37297 } } peers { permanent_uuid: "7d0056b98dac41e1b5044e476d03feaa" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 42717 } } peers { permanent_uuid: "46c3b1dd180748369dd5fb5be8fda535" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43681 } }
I20260925 04:48:21.061599 19371 heartbeater.cc:503] Master 127.0.98.126:36323 was elected leader, sending a full tablet report...
I20260925 04:48:21.061733 19519 ts_tablet_manager.cc:1428] T 036da4f3cef94499ae4b3d9b8ab82e8e P 46c3b1dd180748369dd5fb5be8fda535: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260925 04:48:21.061844 19520 ts_tablet_manager.cc:1428] T 036da4f3cef94499ae4b3d9b8ab82e8e P 7d0056b98dac41e1b5044e476d03feaa: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 04:48:21.062095 19240 heartbeater.cc:503] Master 127.0.98.126:36323 was elected leader, sending a full tablet report...
I20260925 04:48:21.065562 19522 raft_consensus.cc:348] T 036da4f3cef94499ae4b3d9b8ab82e8e P 0f1e0204b7b14908b28c0c4d26d19b25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f1e0204b7b14908b28c0c4d26d19b25" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 37297 } } peers { permanent_uuid: "7d0056b98dac41e1b5044e476d03feaa" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 42717 } } peers { permanent_uuid: "46c3b1dd180748369dd5fb5be8fda535" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43681 } }
I20260925 04:48:21.065861 19522 raft_consensus.cc:374] T 036da4f3cef94499ae4b3d9b8ab82e8e P 0f1e0204b7b14908b28c0c4d26d19b25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:48:21.065963 19522 raft_consensus.cc:729] T 036da4f3cef94499ae4b3d9b8ab82e8e P 0f1e0204b7b14908b28c0c4d26d19b25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f1e0204b7b14908b28c0c4d26d19b25, State: Initialized, Role: FOLLOWER
I20260925 04:48:21.066480 19522 consensus_queue.cc:260] T 036da4f3cef94499ae4b3d9b8ab82e8e P 0f1e0204b7b14908b28c0c4d26d19b25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f1e0204b7b14908b28c0c4d26d19b25" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 37297 } } peers { permanent_uuid: "7d0056b98dac41e1b5044e476d03feaa" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 42717 } } peers { permanent_uuid: "46c3b1dd180748369dd5fb5be8fda535" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43681 } }
I20260925 04:48:21.067135 19502 heartbeater.cc:503] Master 127.0.98.126:36323 was elected leader, sending a full tablet report...
I20260925 04:48:21.067423 19522 ts_tablet_manager.cc:1428] T 036da4f3cef94499ae4b3d9b8ab82e8e P 0f1e0204b7b14908b28c0c4d26d19b25: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
W20260925 04:48:21.109740 19241 tablet.cc:2369] T 036da4f3cef94499ae4b3d9b8ab82e8e P 46c3b1dd180748369dd5fb5be8fda535: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:48:21.170352 19526 raft_consensus.cc:482] T 036da4f3cef94499ae4b3d9b8ab82e8e P 7d0056b98dac41e1b5044e476d03feaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:48:21.170588 19526 raft_consensus.cc:504] T 036da4f3cef94499ae4b3d9b8ab82e8e P 7d0056b98dac41e1b5044e476d03feaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f1e0204b7b14908b28c0c4d26d19b25" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 37297 } } peers { permanent_uuid: "7d0056b98dac41e1b5044e476d03feaa" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 42717 } } peers { permanent_uuid: "46c3b1dd180748369dd5fb5be8fda535" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43681 } }
I20260925 04:48:21.171856 19526 leader_election.cc:288] T 036da4f3cef94499ae4b3d9b8ab82e8e P 7d0056b98dac41e1b5044e476d03feaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f1e0204b7b14908b28c0c4d26d19b25 (127.0.98.67:37297), 46c3b1dd180748369dd5fb5be8fda535 (127.0.98.65:43681)
I20260925 04:48:21.175885 19373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:21.172076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:37297 (local address 127.0.98.66:40639)
0925 04:48:21.172239 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:21.172259 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 04:48:21.172385 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:48:21.172709 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:48:21.172715 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:48:21.172738 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:48:21.172841 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:21.172847 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:21.173715 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:21.173719 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:21.174688 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:21.174697 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:21.175416 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:21.175420 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:21.175543 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:21.175571 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 04:48:21.175677 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 04:48:21.175749 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260925 04:48:21.175884 19515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:21.172233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:40639 (local address 127.0.98.67:37297)
0925 04:48:21.172393 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:21.172398 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:21.172417 (+ 19us) server_negotiation.cc:407] Connection header received
0925 04:48:21.172480 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:21.172483 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:21.172542 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:21.172626 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:48:21.172978 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:21.173600 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:21.174807 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:21.175263 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:21.175401 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:21.175462 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:21.175708 (+ 246us) server_negotiation.cc:299] Negotiation successful
0925 04:48:21.175773 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 04:48:21.175900 19527 raft_consensus.cc:482] T 036da4f3cef94499ae4b3d9b8ab82e8e P 0f1e0204b7b14908b28c0c4d26d19b25 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:48:21.176426 19527 raft_consensus.cc:504] T 036da4f3cef94499ae4b3d9b8ab82e8e P 0f1e0204b7b14908b28c0c4d26d19b25 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f1e0204b7b14908b28c0c4d26d19b25" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 37297 } } peers { permanent_uuid: "7d0056b98dac41e1b5044e476d03feaa" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 42717 } } peers { permanent_uuid: "46c3b1dd180748369dd5fb5be8fda535" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43681 } }
I20260925 04:48:21.176613 19456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "036da4f3cef94499ae4b3d9b8ab82e8e" candidate_uuid: "7d0056b98dac41e1b5044e476d03feaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f1e0204b7b14908b28c0c4d26d19b25" is_pre_election: true
I20260925 04:48:21.176692 19528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:21.172369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:43681 (local address 127.0.98.66:44187)
0925 04:48:21.172773 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:21.172787 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:48:21.172894 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:48:21.173216 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:48:21.173219 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:48:21.173235 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:48:21.173353 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:21.173358 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:21.174208 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:21.174213 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:21.175250 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:21.175260 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:21.176212 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:21.176216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:21.176315 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:21.176337 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 04:48:21.176485 (+ 148us) client_negotiation.cc:240] Negotiation successful
0925 04:48:21.176559 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":160,"threads_started":1}
I20260925 04:48:21.176726 19518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:21.172662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:44187 (local address 127.0.98.65:43681)
0925 04:48:21.172789 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:21.172794 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:21.172866 (+ 72us) server_negotiation.cc:407] Connection header received
0925 04:48:21.173038 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:21.173041 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:21.173092 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:21.173175 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:48:21.173474 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:21.174102 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:21.175429 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:21.175938 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:21.176037 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:21.176102 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:21.176510 (+ 408us) server_negotiation.cc:299] Negotiation successful
0925 04:48:21.176594 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 04:48:21.176875 19456 raft_consensus.cc:2454] T 036da4f3cef94499ae4b3d9b8ab82e8e P 0f1e0204b7b14908b28c0c4d26d19b25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d0056b98dac41e1b5044e476d03feaa in term 0.
I20260925 04:48:21.177348 19261 leader_election.cc:302] T 036da4f3cef94499ae4b3d9b8ab82e8e P 7d0056b98dac41e1b5044e476d03feaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f1e0204b7b14908b28c0c4d26d19b25, 7d0056b98dac41e1b5044e476d03feaa; no voters:
I20260925 04:48:21.177356 19194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "036da4f3cef94499ae4b3d9b8ab82e8e" candidate_uuid: "7d0056b98dac41e1b5044e476d03feaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46c3b1dd180748369dd5fb5be8fda535" is_pre_election: true
I20260925 04:48:21.177619 19194 raft_consensus.cc:2454] T 036da4f3cef94499ae4b3d9b8ab82e8e P 46c3b1dd180748369dd5fb5be8fda535 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d0056b98dac41e1b5044e476d03feaa in term 0.
I20260925 04:48:21.177634 19526 raft_consensus.cc:2781] T 036da4f3cef94499ae4b3d9b8ab82e8e P 7d0056b98dac41e1b5044e476d03feaa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:48:21.177729 19526 raft_consensus.cc:482] T 036da4f3cef94499ae4b3d9b8ab82e8e P 7d0056b98dac41e1b5044e476d03feaa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:48:21.177785 19526 raft_consensus.cc:3037] T 036da4f3cef94499ae4b3d9b8ab82e8e P 7d0056b98dac41e1b5044e476d03feaa [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:48:21.178511 19527 leader_election.cc:288] T 036da4f3cef94499ae4b3d9b8ab82e8e P 0f1e0204b7b14908b28c0c4d26d19b25 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d0056b98dac41e1b5044e476d03feaa (127.0.98.66:42717), 46c3b1dd180748369dd5fb5be8fda535 (127.0.98.65:43681)
I20260925 04:48:21.179196 19526 raft_consensus.cc:504] T 036da4f3cef94499ae4b3d9b8ab82e8e P 7d0056b98dac41e1b5044e476d03feaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f1e0204b7b14908b28c0c4d26d19b25" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 37297 } } peers { permanent_uuid: "7d0056b98dac41e1b5044e476d03feaa" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 42717 } } peers { permanent_uuid: "46c3b1dd180748369dd5fb5be8fda535" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43681 } }
I20260925 04:48:21.179550 19526 leader_election.cc:288] T 036da4f3cef94499ae4b3d9b8ab82e8e P 7d0056b98dac41e1b5044e476d03feaa [CANDIDATE]: Term 1 election: Requested vote from peers 0f1e0204b7b14908b28c0c4d26d19b25 (127.0.98.67:37297), 46c3b1dd180748369dd5fb5be8fda535 (127.0.98.65:43681)
I20260925 04:48:21.179942 19194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "036da4f3cef94499ae4b3d9b8ab82e8e" candidate_uuid: "7d0056b98dac41e1b5044e476d03feaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46c3b1dd180748369dd5fb5be8fda535"
I20260925 04:48:21.180070 19194 raft_consensus.cc:3037] T 036da4f3cef94499ae4b3d9b8ab82e8e P 46c3b1dd180748369dd5fb5be8fda535 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:48:21.180717 19456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "036da4f3cef94499ae4b3d9b8ab82e8e" candidate_uuid: "7d0056b98dac41e1b5044e476d03feaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f1e0204b7b14908b28c0c4d26d19b25"
I20260925 04:48:21.180833 19456 raft_consensus.cc:3037] T 036da4f3cef94499ae4b3d9b8ab82e8e P 0f1e0204b7b14908b28c0c4d26d19b25 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:48:21.181401 19194 raft_consensus.cc:2454] T 036da4f3cef94499ae4b3d9b8ab82e8e P 46c3b1dd180748369dd5fb5be8fda535 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d0056b98dac41e1b5044e476d03feaa in term 1.
I20260925 04:48:21.181749 19261 leader_election.cc:302] T 036da4f3cef94499ae4b3d9b8ab82e8e P 7d0056b98dac41e1b5044e476d03feaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46c3b1dd180748369dd5fb5be8fda535, 7d0056b98dac41e1b5044e476d03feaa; no voters:
I20260925 04:48:21.181929 19526 raft_consensus.cc:2781] T 036da4f3cef94499ae4b3d9b8ab82e8e P 7d0056b98dac41e1b5044e476d03feaa [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:48:21.182164 19526 raft_consensus.cc:686] T 036da4f3cef94499ae4b3d9b8ab82e8e P 7d0056b98dac41e1b5044e476d03feaa [term 1 LEADER]: Becoming Leader. State: Replica: 7d0056b98dac41e1b5044e476d03feaa, State: Running, Role: LEADER
I20260925 04:48:21.182221 19456 raft_consensus.cc:2454] T 036da4f3cef94499ae4b3d9b8ab82e8e P 0f1e0204b7b14908b28c0c4d26d19b25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d0056b98dac41e1b5044e476d03feaa in term 1.
I20260925 04:48:21.182552 19526 consensus_queue.cc:237] T 036da4f3cef94499ae4b3d9b8ab82e8e P 7d0056b98dac41e1b5044e476d03feaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f1e0204b7b14908b28c0c4d26d19b25" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 37297 } } peers { permanent_uuid: "7d0056b98dac41e1b5044e476d03feaa" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 42717 } } peers { permanent_uuid: "46c3b1dd180748369dd5fb5be8fda535" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43681 } }
I20260925 04:48:21.183264 19518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:21.179077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:51897 (local address 127.0.98.65:43681)
0925 04:48:21.179189 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:21.179196 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:21.179209 (+ 13us) server_negotiation.cc:407] Connection header received
0925 04:48:21.179314 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:21.179320 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:21.179361 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:21.179431 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:48:21.179861 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:21.180449 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:21.182120 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:21.182626 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:21.182755 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:21.182812 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:21.183089 (+ 277us) server_negotiation.cc:299] Negotiation successful
0925 04:48:21.183153 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 04:48:21.183291 19529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:21.178773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:43681 (local address 127.0.98.67:51897)
0925 04:48:21.179137 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:21.179150 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:48:21.179237 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:48:21.179444 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:48:21.179447 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:48:21.179463 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:48:21.179728 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:21.179740 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:21.180564 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:21.180591 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:21.181962 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:21.181972 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:21.182863 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:21.182866 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:21.182955 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:21.182974 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 04:48:21.183071 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 04:48:21.183142 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"mutex_wait_us":183,"spinlock_wait_cycles":2304,"thread_start_us":137,"threads_started":1}
I20260925 04:48:21.183863 19194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "036da4f3cef94499ae4b3d9b8ab82e8e" candidate_uuid: "0f1e0204b7b14908b28c0c4d26d19b25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46c3b1dd180748369dd5fb5be8fda535" is_pre_election: true
I20260925 04:48:21.184048 19194 raft_consensus.cc:2379] T 036da4f3cef94499ae4b3d9b8ab82e8e P 46c3b1dd180748369dd5fb5be8fda535 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0f1e0204b7b14908b28c0c4d26d19b25 in current term 1: Already voted for candidate 7d0056b98dac41e1b5044e476d03feaa in this term.
I20260925 04:48:21.184083 19504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:21.178639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:42717 (local address 127.0.98.67:38849)
0925 04:48:21.178776 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:21.178791 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:48:21.178869 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:48:21.179486 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:48:21.179489 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:48:21.179533 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:48:21.179677 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:21.179684 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:21.180626 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:21.180629 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:21.181927 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:21.181938 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:21.183655 (+ 1717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:21.183659 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:21.183772 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:21.183795 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 04:48:21.183874 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 04:48:21.183954 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":50,"spinlock_wait_cycles":3584}
I20260925 04:48:21.184149 19516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:21.179090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:38849 (local address 127.0.98.66:42717)
0925 04:48:21.179208 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:21.179214 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:21.179229 (+ 15us) server_negotiation.cc:407] Connection header received
0925 04:48:21.179265 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:21.179268 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:21.179313 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:21.179387 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:48:21.179860 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:21.180512 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:21.182787 (+ 2275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:21.183316 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:21.183419 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:21.183466 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:21.183910 (+ 444us) server_negotiation.cc:299] Negotiation successful
0925 04:48:21.184006 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 04:48:21.184725 19325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "036da4f3cef94499ae4b3d9b8ab82e8e" candidate_uuid: "0f1e0204b7b14908b28c0c4d26d19b25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d0056b98dac41e1b5044e476d03feaa" is_pre_election: true
I20260925 04:48:21.185351 19389 leader_election.cc:302] T 036da4f3cef94499ae4b3d9b8ab82e8e P 0f1e0204b7b14908b28c0c4d26d19b25 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0f1e0204b7b14908b28c0c4d26d19b25; no voters: 46c3b1dd180748369dd5fb5be8fda535, 7d0056b98dac41e1b5044e476d03feaa
I20260925 04:48:21.185626 19527 raft_consensus.cc:2726] T 036da4f3cef94499ae4b3d9b8ab82e8e P 0f1e0204b7b14908b28c0c4d26d19b25 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 04:48:21.187815 19052 catalog_manager.cc:5495] T 036da4f3cef94499ae4b3d9b8ab82e8e P 7d0056b98dac41e1b5044e476d03feaa reported cstate change: term changed from 0 to 1, leader changed from <none> to 7d0056b98dac41e1b5044e476d03feaa (127.0.98.66). New cstate: current_term: 1 leader_uuid: "7d0056b98dac41e1b5044e476d03feaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f1e0204b7b14908b28c0c4d26d19b25" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 37297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d0056b98dac41e1b5044e476d03feaa" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 42717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46c3b1dd180748369dd5fb5be8fda535" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43681 } health_report { overall_health: UNKNOWN } } }
I20260925 04:48:21.230752 19516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:21.227248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35556 (local address 127.0.98.66:42717)
0925 04:48:21.227387 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:21.227391 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:21.227405 (+ 14us) server_negotiation.cc:407] Connection header received
0925 04:48:21.227491 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:21.227493 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:21.227549 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:21.227643 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:48:21.227880 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:21.228394 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:21.229244 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:21.229463 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:21.229550 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:21.229833 (+ 283us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:48:21.230329 (+ 496us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:48:21.230458 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:21.230566 (+ 108us) server_negotiation.cc:299] Negotiation successful
0925 04:48:21.230599 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 04:48:21.237084 19194 raft_consensus.cc:1261] T 036da4f3cef94499ae4b3d9b8ab82e8e P 46c3b1dd180748369dd5fb5be8fda535 [term 1 FOLLOWER]: Refusing update from remote peer 7d0056b98dac41e1b5044e476d03feaa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:48:21.237123 19456 raft_consensus.cc:1261] T 036da4f3cef94499ae4b3d9b8ab82e8e P 0f1e0204b7b14908b28c0c4d26d19b25 [term 1 FOLLOWER]: Refusing update from remote peer 7d0056b98dac41e1b5044e476d03feaa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:48:21.237808 19526 consensus_queue.cc:1035] T 036da4f3cef94499ae4b3d9b8ab82e8e P 7d0056b98dac41e1b5044e476d03feaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f1e0204b7b14908b28c0c4d26d19b25" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 37297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:48:21.238058 19530 consensus_queue.cc:1035] T 036da4f3cef94499ae4b3d9b8ab82e8e P 7d0056b98dac41e1b5044e476d03feaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "46c3b1dd180748369dd5fb5be8fda535" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 04:48:21.242910 19503 tablet.cc:2369] T 036da4f3cef94499ae4b3d9b8ab82e8e P 0f1e0204b7b14908b28c0c4d26d19b25: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:48:21.247228 19534 mvcc.cc:205] Tried to move back new op lower bound from 7333116728258453504 to 7333116728053563392. Current Snapshot: MvccSnapshot[applied={T|T < 7333116728258453504}]
I20260925 04:48:21.247357 19535 mvcc.cc:205] Tried to move back new op lower bound from 7333116728258453504 to 7333116728053563392. Current Snapshot: MvccSnapshot[applied={T|T < 7333116728258453504}]
I20260925 04:48:21.253523 19052 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:48:21.253708 19052 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:48:21.256377 19052 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 04540d34205f41149b437abb149f3f8c: Sending DeleteTablet for 3 replicas of tablet 036da4f3cef94499ae4b3d9b8ab82e8e
I20260925 04:48:21.257200 19436 tablet_service.cc:1514] Processing DeleteTablet for tablet 036da4f3cef94499ae4b3d9b8ab82e8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:48:21 UTC) from {username='slave'} at 127.0.0.1:37962
I20260925 04:48:21.257328 19174 tablet_service.cc:1514] Processing DeleteTablet for tablet 036da4f3cef94499ae4b3d9b8ab82e8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:48:21 UTC) from {username='slave'} at 127.0.0.1:56152
I20260925 04:48:21.257761 19542 tablet_replica.cc:330] T 036da4f3cef94499ae4b3d9b8ab82e8e P 0f1e0204b7b14908b28c0c4d26d19b25: stopping tablet replica
I20260925 04:48:21.257838 19305 tablet_service.cc:1514] Processing DeleteTablet for tablet 036da4f3cef94499ae4b3d9b8ab82e8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:48:21 UTC) from {username='slave'} at 127.0.0.1:35544
I20260925 04:48:21.258083 19542 raft_consensus.cc:2229] T 036da4f3cef94499ae4b3d9b8ab82e8e P 0f1e0204b7b14908b28c0c4d26d19b25 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:48:21.258152 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 19112
I20260925 04:48:21.258421 19542 raft_consensus.cc:2258] T 036da4f3cef94499ae4b3d9b8ab82e8e P 0f1e0204b7b14908b28c0c4d26d19b25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:48:21.258888 19544 tablet_replica.cc:330] T 036da4f3cef94499ae4b3d9b8ab82e8e P 7d0056b98dac41e1b5044e476d03feaa: stopping tablet replica
I20260925 04:48:21.259231 19544 raft_consensus.cc:2229] T 036da4f3cef94499ae4b3d9b8ab82e8e P 7d0056b98dac41e1b5044e476d03feaa [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:48:21.259621 19544 raft_consensus.cc:2258] T 036da4f3cef94499ae4b3d9b8ab82e8e P 7d0056b98dac41e1b5044e476d03feaa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:48:21.261065 19544 ts_tablet_manager.cc:1905] T 036da4f3cef94499ae4b3d9b8ab82e8e P 7d0056b98dac41e1b5044e476d03feaa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:48:21.261065 19542 ts_tablet_manager.cc:1905] T 036da4f3cef94499ae4b3d9b8ab82e8e P 0f1e0204b7b14908b28c0c4d26d19b25: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:48:21.263885 19544 ts_tablet_manager.cc:1918] T 036da4f3cef94499ae4b3d9b8ab82e8e P 7d0056b98dac41e1b5044e476d03feaa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:48:21.264035 19542 ts_tablet_manager.cc:1918] T 036da4f3cef94499ae4b3d9b8ab82e8e P 0f1e0204b7b14908b28c0c4d26d19b25: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:48:21.264055 19544 log.cc:1196] T 036da4f3cef94499ae4b3d9b8ab82e8e P 7d0056b98dac41e1b5044e476d03feaa: Deleting WAL directory at /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311449873301-393-0/minicluster-data/ts-1/wal/wals/036da4f3cef94499ae4b3d9b8ab82e8e
I20260925 04:48:21.264125 19542 log.cc:1196] T 036da4f3cef94499ae4b3d9b8ab82e8e P 0f1e0204b7b14908b28c0c4d26d19b25: Deleting WAL directory at /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790311449873301-393-0/minicluster-data/ts-2/wal/wals/036da4f3cef94499ae4b3d9b8ab82e8e
I20260925 04:48:21.264474 19542 ts_tablet_manager.cc:1939] T 036da4f3cef94499ae4b3d9b8ab82e8e P 0f1e0204b7b14908b28c0c4d26d19b25: Deleting consensus metadata
I20260925 04:48:21.264503 19544 ts_tablet_manager.cc:1939] T 036da4f3cef94499ae4b3d9b8ab82e8e P 7d0056b98dac41e1b5044e476d03feaa: Deleting consensus metadata
I20260925 04:48:21.265758 19039 catalog_manager.cc:4841] TS 0f1e0204b7b14908b28c0c4d26d19b25 (127.0.98.67:37297): tablet 036da4f3cef94499ae4b3d9b8ab82e8e (table test-table [id=0a56915787b64bd08b18adef2ecf9f77]) successfully deleted
I20260925 04:48:21.265750 19036 catalog_manager.cc:4841] TS 7d0056b98dac41e1b5044e476d03feaa (127.0.98.66:42717): tablet 036da4f3cef94499ae4b3d9b8ab82e8e (table test-table [id=0a56915787b64bd08b18adef2ecf9f77]) successfully deleted
W20260925 04:48:21.267771 19039 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.65:43681 (error 108)
I20260925 04:48:21.268574 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 19243
I20260925 04:48:21.268793 19514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:21.268499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:43681 (local address 127.0.0.1:56158)
0925 04:48:21.268648 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:21.268714 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.65:43681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260925 04:48:21.269444 19039 catalog_manager.cc:4584] TS 46c3b1dd180748369dd5fb5be8fda535 (127.0.98.65:43681): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 036da4f3cef94499ae4b3d9b8ab82e8e: Network error: Client connection negotiation failed: client connection to 127.0.98.65:43681: connect: Connection refused (error 111)
I20260925 04:48:21.274380 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 19374
I20260925 04:48:21.280081 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 19021
2026-09-25T04:48:21Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1139 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:48:21 dist-test-slave-kghz krb5kdc[19548](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:48:21 dist-test-slave-kghz krb5kdc[19548](info): set up 2 sockets
Sep 25 04:48:21 dist-test-slave-kghz krb5kdc[19548](info): commencing operation
krb5kdc: starting...
W20260925 04:48:22.314641 393 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/krb5kdc/test-user.keytab.
Sep 25 04:48:22 dist-test-slave-kghz krb5kdc[19548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:48:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:48:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260925 04:48:22.442606 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:41631
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:39487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:41631
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:48:22.555480 19561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:48:22.555794 19561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:48:22.555881 19561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:48:22.559966 19561 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:48:22.560047 19561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:48:22.560084 19561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:48:22.560139 19561 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:48:22.560190 19561 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:48:22.565524 19561 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:39487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:41631
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:41631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19561
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:48:22.566701 19561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:48:22.567723 19561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:48:22.576355 19567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:48:22.576360 19566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:48:22.576395 19569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:48:22.576675 19561 server_base.cc:1034] running on GCE node
I20260925 04:48:22.577307 19561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:48:22.578617 19561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:48:22.579805 19561 hybrid_clock.cc:648] HybridClock initialized: now 1790311702579783 us; error 43 us; skew 500 ppm
Sep 25 04:48:22 dist-test-slave-kghz krb5kdc[19548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311702, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:48:22.582660 19561 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260925 04:48:22.583591 19561 webserver.cc:460] Webserver started at http://127.0.98.126:33029/ using document root <none> and password file <none>
I20260925 04:48:22.584167 19561 fs_manager.cc:359] Metadata directory not provided
I20260925 04:48:22.584228 19561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:48:22.584447 19561 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:48:22.586640 19561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/minicluster-data/master-0/data/instance:
uuid: "842e16d48f974f1da0c0d3c40548fdc4"
format_stamp: "Formatted at 2026-09-25 04:48:22 on dist-test-slave-kghz"
server_key: "8fe176035abf852cc8d78d6e4fa9be39"
server_key_iv: "8f105e98da1cabd24408c1ebd9ac7095"
server_key_version: "encryptionkey@0"
I20260925 04:48:22.587214 19561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/minicluster-data/master-0/wal/instance:
uuid: "842e16d48f974f1da0c0d3c40548fdc4"
format_stamp: "Formatted at 2026-09-25 04:48:22 on dist-test-slave-kghz"
server_key: "8fe176035abf852cc8d78d6e4fa9be39"
server_key_iv: "8f105e98da1cabd24408c1ebd9ac7095"
server_key_version: "encryptionkey@0"
I20260925 04:48:22.591197 19561 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 04:48:22.593845 19576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:48:22.595239 19561 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:48:22.595393 19561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/minicluster-data/master-0/wal
uuid: "842e16d48f974f1da0c0d3c40548fdc4"
format_stamp: "Formatted at 2026-09-25 04:48:22 on dist-test-slave-kghz"
server_key: "8fe176035abf852cc8d78d6e4fa9be39"
server_key_iv: "8f105e98da1cabd24408c1ebd9ac7095"
server_key_version: "encryptionkey@0"
I20260925 04:48:22.595530 19561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:48:22.612727 19561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:48:22.616178 19561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:48:22.616397 19561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:48:22.623660 19561 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:41631
I20260925 04:48:22.623685 19628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:41631 every 8 connection(s)
I20260925 04:48:22.624625 19561 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
I20260925 04:48:22.627434 19629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:48:22.628973 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 19561
I20260925 04:48:22.629097 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/minicluster-data/master-0/wal/instance
I20260925 04:48:22.629410 393 external_mini_cluster.cc:1428] Setting key a5cb5c297095af06e2fda74465839413
I20260925 04:48:22.633570 19629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 842e16d48f974f1da0c0d3c40548fdc4: Bootstrap starting.
I20260925 04:48:22.635885 19629 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 842e16d48f974f1da0c0d3c40548fdc4: Neither blocks nor log segments found. Creating new log.
I20260925 04:48:22.636682 19629 log.cc:824] T 00000000000000000000000000000000 P 842e16d48f974f1da0c0d3c40548fdc4: Log is configured to *not* fsync() on all Append() calls
Sep 25 04:48:22 dist-test-slave-kghz krb5kdc[19548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:48:22.638959 19629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 842e16d48f974f1da0c0d3c40548fdc4: No bootstrap required, opened a new log
I20260925 04:48:22.643839 19629 raft_consensus.cc:348] T 00000000000000000000000000000000 P 842e16d48f974f1da0c0d3c40548fdc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "842e16d48f974f1da0c0d3c40548fdc4" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41631 } }
I20260925 04:48:22.644121 19629 raft_consensus.cc:374] T 00000000000000000000000000000000 P 842e16d48f974f1da0c0d3c40548fdc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:48:22.644174 19629 raft_consensus.cc:729] T 00000000000000000000000000000000 P 842e16d48f974f1da0c0d3c40548fdc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 842e16d48f974f1da0c0d3c40548fdc4, State: Initialized, Role: FOLLOWER
I20260925 04:48:22.644644 19632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:22.630902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36340 (local address 127.0.98.126:41631)
0925 04:48:22.631349 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:22.631370 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:22.631412 (+ 42us) server_negotiation.cc:407] Connection header received
0925 04:48:22.632070 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:22.632088 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:22.632411 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:22.632759 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:22.633557 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:22.634649 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:22.635538 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:22.636083 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:22.636258 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:22.639184 (+ 2926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:22.639191 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:22.639216 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:48:22.639246 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:22.641358 (+ 2112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:22.641828 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:22.641834 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:22.641840 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:22.641978 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:22.642359 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:22.642363 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:22.642365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:22.642881 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:22.643092 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:22.643373 (+ 281us) server_negotiation.cc:299] Negotiation successful
0925 04:48:22.643670 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":138,"threads_started":1}
I20260925 04:48:22.644654 19629 consensus_queue.cc:260] T 00000000000000000000000000000000 P 842e16d48f974f1da0c0d3c40548fdc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "842e16d48f974f1da0c0d3c40548fdc4" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41631 } }
I20260925 04:48:22.644831 19629 raft_consensus.cc:388] T 00000000000000000000000000000000 P 842e16d48f974f1da0c0d3c40548fdc4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:48:22.644904 19629 raft_consensus.cc:482] T 00000000000000000000000000000000 P 842e16d48f974f1da0c0d3c40548fdc4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:48:22.645004 19629 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 842e16d48f974f1da0c0d3c40548fdc4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:48:22.646131 19629 raft_consensus.cc:504] T 00000000000000000000000000000000 P 842e16d48f974f1da0c0d3c40548fdc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "842e16d48f974f1da0c0d3c40548fdc4" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41631 } }
I20260925 04:48:22.646503 19629 leader_election.cc:302] T 00000000000000000000000000000000 P 842e16d48f974f1da0c0d3c40548fdc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 842e16d48f974f1da0c0d3c40548fdc4; no voters:
I20260925 04:48:22.646858 19629 leader_election.cc:288] T 00000000000000000000000000000000 P 842e16d48f974f1da0c0d3c40548fdc4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:48:22.646981 19634 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 842e16d48f974f1da0c0d3c40548fdc4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:48:22.647284 19634 raft_consensus.cc:686] T 00000000000000000000000000000000 P 842e16d48f974f1da0c0d3c40548fdc4 [term 1 LEADER]: Becoming Leader. State: Replica: 842e16d48f974f1da0c0d3c40548fdc4, State: Running, Role: LEADER
I20260925 04:48:22.647642 19634 consensus_queue.cc:237] T 00000000000000000000000000000000 P 842e16d48f974f1da0c0d3c40548fdc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "842e16d48f974f1da0c0d3c40548fdc4" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41631 } }
I20260925 04:48:22.648139 19629 sys_catalog.cc:564] T 00000000000000000000000000000000 P 842e16d48f974f1da0c0d3c40548fdc4 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:48:22.649672 19635 sys_catalog.cc:455] T 00000000000000000000000000000000 P 842e16d48f974f1da0c0d3c40548fdc4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "842e16d48f974f1da0c0d3c40548fdc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "842e16d48f974f1da0c0d3c40548fdc4" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41631 } } }
I20260925 04:48:22.649823 19635 sys_catalog.cc:458] T 00000000000000000000000000000000 P 842e16d48f974f1da0c0d3c40548fdc4 [sys.catalog]: This master's current role is: LEADER
I20260925 04:48:22.650666 19636 sys_catalog.cc:455] T 00000000000000000000000000000000 P 842e16d48f974f1da0c0d3c40548fdc4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 842e16d48f974f1da0c0d3c40548fdc4. Latest consensus state: current_term: 1 leader_uuid: "842e16d48f974f1da0c0d3c40548fdc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "842e16d48f974f1da0c0d3c40548fdc4" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41631 } } }
I20260925 04:48:22.650862 19636 sys_catalog.cc:458] T 00000000000000000000000000000000 P 842e16d48f974f1da0c0d3c40548fdc4 [sys.catalog]: This master's current role is: LEADER
I20260925 04:48:22.651029 19643 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:48:22.654457 19643 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:48:22.659448 19643 catalog_manager.cc:1347] Generated new cluster ID: 0f232b1436004adc932ed03cb06bc1e2
I20260925 04:48:22.659523 19643 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:48:22.672693 19643 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:48:22.673991 19643 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:48:22.682807 19643 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 842e16d48f974f1da0c0d3c40548fdc4: Generated new TSK 0
I20260925 04:48:22.683643 19643 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/krb5kdc/kudu_127.0.98.65.keytab.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/krb5kdc/kudu_127.0.98.65.keytab.
I20260925 04:48:22.714231 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:0
--local_ip_for_outbound_sockets=127.0.98.65
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:41631
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:39487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:48:22.825767 19655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:48:22.826050 19655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:48:22.826093 19655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:48:22.830157 19655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:48:22.830219 19655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:48:22.830293 19655 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260925 04:48:22.835650 19655 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:39487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:41631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:48:22.836750 19655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:48:22.837857 19655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:48:22.847571 19661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:48:22.847560 19663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:48:22.847571 19660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:48:22.847746 19655 server_base.cc:1034] running on GCE node
I20260925 04:48:22.848320 19655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:48:22.848991 19655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:48:22.850203 19655 hybrid_clock.cc:648] HybridClock initialized: now 1790311702850167 us; error 47 us; skew 500 ppm
Sep 25 04:48:22 dist-test-slave-kghz krb5kdc[19548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311702, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:48:22.853036 19655 init.cc:377] Logged in from keytab as kudu/127.0.98.65@KRBTEST.COM (short username kudu)
I20260925 04:48:22.853964 19655 webserver.cc:460] Webserver started at http://127.0.98.65:36053/ using document root <none> and password file <none>
I20260925 04:48:22.854600 19655 fs_manager.cc:359] Metadata directory not provided
I20260925 04:48:22.854701 19655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:48:22.854938 19655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:48:22.856997 19655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/minicluster-data/ts-0/data/instance:
uuid: "9dcaef5b60b94a65ae3b0742ac14e43d"
format_stamp: "Formatted at 2026-09-25 04:48:22 on dist-test-slave-kghz"
server_key: "009c5dc549959dfb64abf21704472f8c"
server_key_iv: "2674aaed4880e1cfbe4590dfb4d5db91"
server_key_version: "encryptionkey@0"
I20260925 04:48:22.857594 19655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/minicluster-data/ts-0/wal/instance:
uuid: "9dcaef5b60b94a65ae3b0742ac14e43d"
format_stamp: "Formatted at 2026-09-25 04:48:22 on dist-test-slave-kghz"
server_key: "009c5dc549959dfb64abf21704472f8c"
server_key_iv: "2674aaed4880e1cfbe4590dfb4d5db91"
server_key_version: "encryptionkey@0"
I20260925 04:48:22.861410 19655 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260925 04:48:22.863974 19670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:48:22.865195 19655 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 04:48:22.865336 19655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/minicluster-data/ts-0/wal
uuid: "9dcaef5b60b94a65ae3b0742ac14e43d"
format_stamp: "Formatted at 2026-09-25 04:48:22 on dist-test-slave-kghz"
server_key: "009c5dc549959dfb64abf21704472f8c"
server_key_iv: "2674aaed4880e1cfbe4590dfb4d5db91"
server_key_version: "encryptionkey@0"
I20260925 04:48:22.865458 19655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:48:22.886540 19655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:48:22.889670 19655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:48:22.889904 19655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:48:22.890619 19655 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:48:22.891453 19655 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:48:22.891523 19655 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:48:22.891595 19655 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:48:22.891647 19655 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:48:22.900842 19655 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:43865
I20260925 04:48:22.900904 19783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:43865 every 8 connection(s)
I20260925 04:48:22.901768 19655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
I20260925 04:48:22.910993 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 19655
I20260925 04:48:22.911123 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/minicluster-data/ts-0/wal/instance
I20260925 04:48:22.911419 393 external_mini_cluster.cc:1428] Setting key 2ab677ef63bfb7d14e81d83d2e6d05a6
Sep 25 04:48:22 dist-test-slave-kghz krb5kdc[19548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311702, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:48:22.916491 19632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:22.903751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:58077 (local address 127.0.98.126:41631)
0925 04:48:22.903919 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:22.903925 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:22.904794 (+ 869us) server_negotiation.cc:407] Connection header received
0925 04:48:22.906005 (+ 1211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:22.906008 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:22.906060 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:22.906148 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:22.907375 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:22.907919 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:22.909030 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:22.909272 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:22.909360 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:22.912273 (+ 2913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:22.912277 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:22.912279 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:48:22.912304 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:22.913956 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:22.914632 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:22.914635 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:22.914636 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:22.914711 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:22.915160 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:22.915165 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:22.915167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:22.915435 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:22.915541 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:22.916087 (+ 546us) server_negotiation.cc:299] Negotiation successful
0925 04:48:22.916229 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 04:48:22.917122 19786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:22.904172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:41631 (local address 127.0.98.65:58077)
0925 04:48:22.904598 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:22.904643 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 04:48:22.905785 (+ 1142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:48:22.906377 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:48:22.906384 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:48:22.906739 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:48:22.907207 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:22.907220 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:22.908032 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:22.908034 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:22.908911 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:22.908919 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:22.909418 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:22.909421 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:22.909565 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:22.910281 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:48:22.910312 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:48:22.912071 (+ 1759us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:48:22.914082 (+ 2011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:48:22.914091 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:48:22.914110 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:48:22.914534 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:48:22.914836 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:48:22.914841 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:48:22.914844 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:48:22.915045 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:48:22.915551 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:48:22.915556 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:48:22.915831 (+ 275us) client_negotiation.cc:769] Sending connection context
0925 04:48:22.915987 (+ 156us) client_negotiation.cc:240] Negotiation successful
0925 04:48:22.916242 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":99,"threads_started":1}
I20260925 04:48:22.918453 19784 heartbeater.cc:347] Connected to a master server at 127.0.98.126:41631
I20260925 04:48:22.918702 19784 heartbeater.cc:464] Registering TS with master...
I20260925 04:48:22.919188 19784 heartbeater.cc:511] Master 127.0.98.126:41631 requested a full tablet report, sending...
I20260925 04:48:22.920924 19593 ts_manager.cc:194] Registered new tserver with Master: 9dcaef5b60b94a65ae3b0742ac14e43d (127.0.98.65:43865)
WARNING: no policy specified for kudu/127.0.98.66@KRBTEST.COM; defaulting to no policy
I20260925 04:48:22.922652 19593 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.65@KRBTEST.COM'} at 127.0.98.65:58077
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/krb5kdc/kudu_127.0.98.66.keytab.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/krb5kdc/kudu_127.0.98.66.keytab.
I20260925 04:48:22.940909 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:0
--local_ip_for_outbound_sockets=127.0.98.66
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:41631
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:39487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:48:23.051628 19789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:48:23.051906 19789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:48:23.051950 19789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:48:23.056100 19789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:48:23.056165 19789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:48:23.056267 19789 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260925 04:48:23.061776 19789 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:39487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:41631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19789
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:48:23.062915 19789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:48:23.063967 19789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:48:23.073781 19794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:48:23.073781 19797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:48:23.073894 19789 server_base.cc:1034] running on GCE node
W20260925 04:48:23.073843 19795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:48:23.074587 19789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:48:23.075205 19789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:48:23.076392 19789 hybrid_clock.cc:648] HybridClock initialized: now 1790311703076373 us; error 31 us; skew 500 ppm
Sep 25 04:48:23 dist-test-slave-kghz krb5kdc[19548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311703, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:48:23.079497 19789 init.cc:377] Logged in from keytab as kudu/127.0.98.66@KRBTEST.COM (short username kudu)
I20260925 04:48:23.080439 19789 webserver.cc:460] Webserver started at http://127.0.98.66:45989/ using document root <none> and password file <none>
I20260925 04:48:23.080988 19789 fs_manager.cc:359] Metadata directory not provided
I20260925 04:48:23.081046 19789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:48:23.081213 19789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:48:23.083302 19789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/minicluster-data/ts-1/data/instance:
uuid: "6429e5ed13e64f4db920b64dc0bc253e"
format_stamp: "Formatted at 2026-09-25 04:48:23 on dist-test-slave-kghz"
server_key: "ebc3fc56a603ab3091ffb31e7f274c40"
server_key_iv: "bfc62576a42eacc82e8af69749010a2c"
server_key_version: "encryptionkey@0"
I20260925 04:48:23.083823 19789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/minicluster-data/ts-1/wal/instance:
uuid: "6429e5ed13e64f4db920b64dc0bc253e"
format_stamp: "Formatted at 2026-09-25 04:48:23 on dist-test-slave-kghz"
server_key: "ebc3fc56a603ab3091ffb31e7f274c40"
server_key_iv: "bfc62576a42eacc82e8af69749010a2c"
server_key_version: "encryptionkey@0"
I20260925 04:48:23.087486 19789 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 04:48:23.090479 19804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:48:23.091811 19789 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:48:23.091972 19789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/minicluster-data/ts-1/wal
uuid: "6429e5ed13e64f4db920b64dc0bc253e"
format_stamp: "Formatted at 2026-09-25 04:48:23 on dist-test-slave-kghz"
server_key: "ebc3fc56a603ab3091ffb31e7f274c40"
server_key_iv: "bfc62576a42eacc82e8af69749010a2c"
server_key_version: "encryptionkey@0"
I20260925 04:48:23.092095 19789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:48:23.112376 19789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:48:23.115857 19789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:48:23.116077 19789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:48:23.116708 19789 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:48:23.117542 19789 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:48:23.117611 19789 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:48:23.117697 19789 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:48:23.117741 19789 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:48:23.127261 19789 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:33545
I20260925 04:48:23.127306 19917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:33545 every 8 connection(s)
I20260925 04:48:23.128149 19789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
I20260925 04:48:23.137521 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 19789
I20260925 04:48:23.137688 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/minicluster-data/ts-1/wal/instance
Sep 25 04:48:23 dist-test-slave-kghz krb5kdc[19548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311703, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:48:23.138113 393 external_mini_cluster.cc:1428] Setting key c1e9d67c8c29811abbd59934550d666a
I20260925 04:48:23.142732 19632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:23.130061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:33563 (local address 127.0.98.126:41631)
0925 04:48:23.130202 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:23.130206 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:23.131058 (+ 852us) server_negotiation.cc:407] Connection header received
0925 04:48:23.132331 (+ 1273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:23.132334 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:23.132393 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:23.132475 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:23.133665 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:23.134198 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:23.135399 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:23.135626 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:23.135740 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:23.138665 (+ 2925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:23.138668 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:23.138671 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:48:23.138708 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:23.140344 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:23.140973 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:23.140975 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:23.140976 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:23.141050 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:23.141445 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:23.141447 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:23.141448 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:23.141692 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:23.141807 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:23.142369 (+ 562us) server_negotiation.cc:299] Negotiation successful
0925 04:48:23.142516 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 04:48:23.143816 19920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:23.130442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:41631 (local address 127.0.98.66:33563)
0925 04:48:23.130900 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:23.130942 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 04:48:23.132116 (+ 1174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:48:23.132640 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:48:23.132647 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:48:23.133001 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:48:23.133514 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:23.133527 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:23.134381 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:23.134384 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:23.135269 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:23.135278 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:23.135796 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:23.135800 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:23.135965 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:23.136578 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:48:23.136594 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:48:23.138397 (+ 1803us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:48:23.140465 (+ 2068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:48:23.140473 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:48:23.140492 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:48:23.140872 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:48:23.141140 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:48:23.141146 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:48:23.141150 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:48:23.141346 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:48:23.141822 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:48:23.141829 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:48:23.142082 (+ 253us) client_negotiation.cc:769] Sending connection context
0925 04:48:23.142305 (+ 223us) client_negotiation.cc:240] Negotiation successful
0925 04:48:23.142645 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":101,"threads_started":1}
I20260925 04:48:23.145053 19918 heartbeater.cc:347] Connected to a master server at 127.0.98.126:41631
I20260925 04:48:23.145299 19918 heartbeater.cc:464] Registering TS with master...
I20260925 04:48:23.145795 19918 heartbeater.cc:511] Master 127.0.98.126:41631 requested a full tablet report, sending...
I20260925 04:48:23.146876 19593 ts_manager.cc:194] Registered new tserver with Master: 6429e5ed13e64f4db920b64dc0bc253e (127.0.98.66:33545)
I20260925 04:48:23.147454 19593 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.66@KRBTEST.COM'} at 127.0.98.66:33563
WARNING: no policy specified for kudu/127.0.98.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/krb5kdc/kudu_127.0.98.67.keytab.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/krb5kdc/kudu_127.0.98.67.keytab.
I20260925 04:48:23.169911 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:0
--local_ip_for_outbound_sockets=127.0.98.67
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:41631
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:39487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:48:23.281747 19923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:48:23.282083 19923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:48:23.282162 19923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:48:23.286366 19923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:48:23.286454 19923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:48:23.286556 19923 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260925 04:48:23.292052 19923 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:39487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:41631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19923
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:48:23.293193 19923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:48:23.294322 19923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:48:23.305078 19931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:48:23.305068 19928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:48:23.305068 19929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:48:23.305121 19923 server_base.cc:1034] running on GCE node
I20260925 04:48:23.305752 19923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:48:23.306478 19923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:48:23.307657 19923 hybrid_clock.cc:648] HybridClock initialized: now 1790311703307640 us; error 30 us; skew 500 ppm
Sep 25 04:48:23 dist-test-slave-kghz krb5kdc[19548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311703, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:48:23.310664 19923 init.cc:377] Logged in from keytab as kudu/127.0.98.67@KRBTEST.COM (short username kudu)
I20260925 04:48:23.311759 19923 webserver.cc:460] Webserver started at http://127.0.98.67:39001/ using document root <none> and password file <none>
I20260925 04:48:23.312356 19923 fs_manager.cc:359] Metadata directory not provided
I20260925 04:48:23.312423 19923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:48:23.312644 19923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:48:23.314935 19923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/minicluster-data/ts-2/data/instance:
uuid: "802977ce134243b4920229004ae84002"
format_stamp: "Formatted at 2026-09-25 04:48:23 on dist-test-slave-kghz"
server_key: "a6431889525e37cafbd5bef46d9676ae"
server_key_iv: "2add27400acbb490a4f04abad0846c3a"
server_key_version: "encryptionkey@0"
I20260925 04:48:23.315521 19923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/minicluster-data/ts-2/wal/instance:
uuid: "802977ce134243b4920229004ae84002"
format_stamp: "Formatted at 2026-09-25 04:48:23 on dist-test-slave-kghz"
server_key: "a6431889525e37cafbd5bef46d9676ae"
server_key_iv: "2add27400acbb490a4f04abad0846c3a"
server_key_version: "encryptionkey@0"
I20260925 04:48:23.319411 19923 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 04:48:23.322077 19938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:48:23.323396 19923 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 04:48:23.323539 19923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/minicluster-data/ts-2/wal
uuid: "802977ce134243b4920229004ae84002"
format_stamp: "Formatted at 2026-09-25 04:48:23 on dist-test-slave-kghz"
server_key: "a6431889525e37cafbd5bef46d9676ae"
server_key_iv: "2add27400acbb490a4f04abad0846c3a"
server_key_version: "encryptionkey@0"
I20260925 04:48:23.323657 19923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:48:23.337905 19923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:48:23.341352 19923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:48:23.341568 19923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:48:23.342170 19923 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:48:23.343057 19923 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:48:23.343132 19923 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:48:23.343204 19923 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:48:23.343247 19923 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:48:23.352828 19923 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:41123
I20260925 04:48:23.352857 20051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:41123 every 8 connection(s)
I20260925 04:48:23.353715 19923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
I20260925 04:48:23.356439 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 19923
I20260925 04:48:23.356572 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/minicluster-data/ts-2/wal/instance
I20260925 04:48:23.356907 393 external_mini_cluster.cc:1428] Setting key 8c6932a378741de0d1ff94de47bc5c84
Sep 25 04:48:23 dist-test-slave-kghz krb5kdc[19548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311703, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:48:23.367635 19632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:23.355647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:44869 (local address 127.0.98.126:41631)
0925 04:48:23.355785 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:23.355788 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:23.356581 (+ 793us) server_negotiation.cc:407] Connection header received
0925 04:48:23.358140 (+ 1559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:23.358143 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:23.358208 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:23.358360 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:23.359753 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:23.360305 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:23.361434 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:23.361660 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:23.361745 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:23.364351 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:23.364354 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:23.364356 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:48:23.364382 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:23.365661 (+ 1279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:23.366155 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:23.366158 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:23.366159 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:23.366232 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:23.366663 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:23.366665 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:23.366666 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:23.366851 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:23.366931 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:23.367417 (+ 486us) server_negotiation.cc:299] Negotiation successful
0925 04:48:23.367513 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 04:48:23.368535 20054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:23.355972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:41631 (local address 127.0.98.67:44869)
0925 04:48:23.356421 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:23.356463 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 04:48:23.357937 (+ 1474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:48:23.358541 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:48:23.358551 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:48:23.359072 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:48:23.359595 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:23.359608 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:23.360420 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:23.360423 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:23.361308 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:23.361317 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:23.361791 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:23.361794 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:23.361952 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:23.362662 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:48:23.362678 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:48:23.364193 (+ 1515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:48:23.365797 (+ 1604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:48:23.365802 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:48:23.365813 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:48:23.366067 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:48:23.366423 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:48:23.366426 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:48:23.366428 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:48:23.366571 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:48:23.366966 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:48:23.366971 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:48:23.367209 (+ 238us) client_negotiation.cc:769] Sending connection context
0925 04:48:23.367378 (+ 169us) client_negotiation.cc:240] Negotiation successful
0925 04:48:23.367649 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":131,"threads_started":1}
I20260925 04:48:23.369836 20052 heartbeater.cc:347] Connected to a master server at 127.0.98.126:41631
I20260925 04:48:23.370105 20052 heartbeater.cc:464] Registering TS with master...
I20260925 04:48:23.370765 20052 heartbeater.cc:511] Master 127.0.98.126:41631 requested a full tablet report, sending...
I20260925 04:48:23.371881 19593 ts_manager.cc:194] Registered new tserver with Master: 802977ce134243b4920229004ae84002 (127.0.98.67:41123)
I20260925 04:48:23.372465 19593 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.67@KRBTEST.COM'} at 127.0.98.67:44869
I20260925 04:48:23.382825 393 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 04:48:23.392349 19632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:23.384774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36348 (local address 127.0.98.126:41631)
0925 04:48:23.384905 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:23.384909 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:23.385122 (+ 213us) server_negotiation.cc:407] Connection header received
0925 04:48:23.385219 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:23.385221 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:23.385260 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:23.385329 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:23.386018 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:23.386554 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:23.387470 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:23.387697 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:23.387806 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:23.389253 (+ 1447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:23.389256 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:23.389258 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:48:23.389283 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:23.390711 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:23.391149 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:23.391152 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:23.391153 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:23.391238 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:23.391524 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:23.391527 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:23.391528 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:23.391779 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:23.391895 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:23.392129 (+ 234us) server_negotiation.cc:299] Negotiation successful
0925 04:48:23.392228 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 04:48:23.395306 19593 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36348:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:48:23.398624 19593 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:48:23.413455 20063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:23.407126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:43865 (local address 127.0.0.1:38056)
0925 04:48:23.408168 (+ 1042us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:23.408182 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:48:23.408286 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:48:23.408996 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:48:23.408998 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:48:23.409031 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:48:23.409445 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:23.409457 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:23.410941 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:23.410945 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:23.412089 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:23.412098 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:23.412874 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:23.412877 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:23.413001 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:23.413041 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 04:48:23.413151 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 04:48:23.413270 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":978,"mutex_wait_us":229,"spinlock_wait_cycles":6528,"thread_start_us":140,"threads_started":1}
I20260925 04:48:23.413623 20067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:23.407855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45846 (local address 127.0.98.66:33545)
0925 04:48:23.408478 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:23.408492 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:23.408517 (+ 25us) server_negotiation.cc:407] Connection header received
0925 04:48:23.408595 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:23.408600 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:23.408822 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:23.408986 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:48:23.409439 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:23.410479 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:23.411901 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:23.412752 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:23.412987 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:23.413128 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:23.413366 (+ 238us) server_negotiation.cc:299] Negotiation successful
0925 04:48:23.413470 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":118,"threads_started":1}
I20260925 04:48:23.413635 20064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:23.407125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:33545 (local address 127.0.0.1:45846)
0925 04:48:23.407587 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:23.407637 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 04:48:23.407824 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:48:23.408990 (+ 1166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:48:23.408995 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:48:23.409029 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:48:23.409285 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:23.409293 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:23.410605 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:23.410611 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:23.411775 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:23.411784 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:23.412886 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:23.412889 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:23.413225 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:23.413242 (+ 17us) client_negotiation.cc:769] Sending connection context
0925 04:48:23.413340 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 04:48:23.413417 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":255,"spinlock_wait_cycles":4480,"thread_start_us":105,"threads_started":1}
I20260925 04:48:23.414547 20065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:23.407373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38056 (local address 127.0.98.65:43865)
0925 04:48:23.408563 (+ 1190us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:23.408577 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:23.408595 (+ 18us) server_negotiation.cc:407] Connection header received
0925 04:48:23.408656 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:23.408660 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:23.408822 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:23.408985 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:48:23.409585 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:23.410779 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:23.412213 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:23.412746 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:23.412882 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:23.413041 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:23.414287 (+ 1246us) server_negotiation.cc:299] Negotiation successful
0925 04:48:23.414388 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1093,"thread_start_us":69,"threads_started":1}
I20260925 04:48:23.414958 20066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:23.407882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:41123 (local address 127.0.0.1:34980)
0925 04:48:23.408750 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:23.408769 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 04:48:23.408869 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:48:23.409823 (+ 954us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:48:23.409826 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:48:23.409880 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:48:23.409995 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:23.410001 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:23.411953 (+ 1952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:23.411958 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:23.413046 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:23.413055 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:23.414558 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:23.414564 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:23.414679 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:23.414707 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 04:48:23.414792 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 04:48:23.414862 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":699,"thread_start_us":102,"threads_started":1}
I20260925 04:48:23.415599 20068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:23.408350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34980 (local address 127.0.98.67:41123)
0925 04:48:23.409320 (+ 970us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:23.409335 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:23.409381 (+ 46us) server_negotiation.cc:407] Connection header received
0925 04:48:23.409455 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:23.409462 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:23.409674 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:23.409831 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:48:23.410134 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:23.411785 (+ 1651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:23.413150 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:23.413978 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:23.414067 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:23.414185 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:23.415326 (+ 1141us) server_negotiation.cc:299] Negotiation successful
0925 04:48:23.415453 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":867,"thread_start_us":184,"threads_started":1}
I20260925 04:48:23.416767 19852 tablet_service.cc:1467] Processing CreateTablet for tablet 2976d2f661c9407691f16a7ee4d6267a (DEFAULT_TABLE table=test-table [id=cbfb30b358a34d27a8ba73d990452c5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:48:23.417026 19718 tablet_service.cc:1467] Processing CreateTablet for tablet 2976d2f661c9407691f16a7ee4d6267a (DEFAULT_TABLE table=test-table [id=cbfb30b358a34d27a8ba73d990452c5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:48:23.417900 19852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2976d2f661c9407691f16a7ee4d6267a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:48:23.418045 19718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2976d2f661c9407691f16a7ee4d6267a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:48:23.419929 19986 tablet_service.cc:1467] Processing CreateTablet for tablet 2976d2f661c9407691f16a7ee4d6267a (DEFAULT_TABLE table=test-table [id=cbfb30b358a34d27a8ba73d990452c5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:48:23.420934 19986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2976d2f661c9407691f16a7ee4d6267a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:48:23.424351 20069 tablet_bootstrap.cc:492] T 2976d2f661c9407691f16a7ee4d6267a P 9dcaef5b60b94a65ae3b0742ac14e43d: Bootstrap starting.
I20260925 04:48:23.425806 20070 tablet_bootstrap.cc:492] T 2976d2f661c9407691f16a7ee4d6267a P 6429e5ed13e64f4db920b64dc0bc253e: Bootstrap starting.
I20260925 04:48:23.427009 20069 tablet_bootstrap.cc:654] T 2976d2f661c9407691f16a7ee4d6267a P 9dcaef5b60b94a65ae3b0742ac14e43d: Neither blocks nor log segments found. Creating new log.
I20260925 04:48:23.427977 20069 log.cc:824] T 2976d2f661c9407691f16a7ee4d6267a P 9dcaef5b60b94a65ae3b0742ac14e43d: Log is configured to *not* fsync() on all Append() calls
I20260925 04:48:23.428030 20071 tablet_bootstrap.cc:492] T 2976d2f661c9407691f16a7ee4d6267a P 802977ce134243b4920229004ae84002: Bootstrap starting.
I20260925 04:48:23.428651 20070 tablet_bootstrap.cc:654] T 2976d2f661c9407691f16a7ee4d6267a P 6429e5ed13e64f4db920b64dc0bc253e: Neither blocks nor log segments found. Creating new log.
I20260925 04:48:23.429371 20070 log.cc:824] T 2976d2f661c9407691f16a7ee4d6267a P 6429e5ed13e64f4db920b64dc0bc253e: Log is configured to *not* fsync() on all Append() calls
I20260925 04:48:23.430013 20071 tablet_bootstrap.cc:654] T 2976d2f661c9407691f16a7ee4d6267a P 802977ce134243b4920229004ae84002: Neither blocks nor log segments found. Creating new log.
I20260925 04:48:23.431171 20071 log.cc:824] T 2976d2f661c9407691f16a7ee4d6267a P 802977ce134243b4920229004ae84002: Log is configured to *not* fsync() on all Append() calls
I20260925 04:48:23.431290 20070 tablet_bootstrap.cc:492] T 2976d2f661c9407691f16a7ee4d6267a P 6429e5ed13e64f4db920b64dc0bc253e: No bootstrap required, opened a new log
I20260925 04:48:23.431489 20070 ts_tablet_manager.cc:1397] T 2976d2f661c9407691f16a7ee4d6267a P 6429e5ed13e64f4db920b64dc0bc253e: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 04:48:23.432451 20069 tablet_bootstrap.cc:492] T 2976d2f661c9407691f16a7ee4d6267a P 9dcaef5b60b94a65ae3b0742ac14e43d: No bootstrap required, opened a new log
I20260925 04:48:23.432595 20069 ts_tablet_manager.cc:1397] T 2976d2f661c9407691f16a7ee4d6267a P 9dcaef5b60b94a65ae3b0742ac14e43d: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260925 04:48:23.434540 20070 raft_consensus.cc:348] T 2976d2f661c9407691f16a7ee4d6267a P 6429e5ed13e64f4db920b64dc0bc253e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcaef5b60b94a65ae3b0742ac14e43d" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43865 } } peers { permanent_uuid: "6429e5ed13e64f4db920b64dc0bc253e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 33545 } } peers { permanent_uuid: "802977ce134243b4920229004ae84002" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 41123 } }
I20260925 04:48:23.434805 20070 raft_consensus.cc:374] T 2976d2f661c9407691f16a7ee4d6267a P 6429e5ed13e64f4db920b64dc0bc253e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:48:23.434894 20070 raft_consensus.cc:729] T 2976d2f661c9407691f16a7ee4d6267a P 6429e5ed13e64f4db920b64dc0bc253e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6429e5ed13e64f4db920b64dc0bc253e, State: Initialized, Role: FOLLOWER
I20260925 04:48:23.435369 20070 consensus_queue.cc:260] T 2976d2f661c9407691f16a7ee4d6267a P 6429e5ed13e64f4db920b64dc0bc253e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcaef5b60b94a65ae3b0742ac14e43d" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43865 } } peers { permanent_uuid: "6429e5ed13e64f4db920b64dc0bc253e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 33545 } } peers { permanent_uuid: "802977ce134243b4920229004ae84002" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 41123 } }
I20260925 04:48:23.436012 19918 heartbeater.cc:503] Master 127.0.98.126:41631 was elected leader, sending a full tablet report...
I20260925 04:48:23.436134 20071 tablet_bootstrap.cc:492] T 2976d2f661c9407691f16a7ee4d6267a P 802977ce134243b4920229004ae84002: No bootstrap required, opened a new log
I20260925 04:48:23.436340 20071 ts_tablet_manager.cc:1397] T 2976d2f661c9407691f16a7ee4d6267a P 802977ce134243b4920229004ae84002: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260925 04:48:23.436561 20069 raft_consensus.cc:348] T 2976d2f661c9407691f16a7ee4d6267a P 9dcaef5b60b94a65ae3b0742ac14e43d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcaef5b60b94a65ae3b0742ac14e43d" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43865 } } peers { permanent_uuid: "6429e5ed13e64f4db920b64dc0bc253e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 33545 } } peers { permanent_uuid: "802977ce134243b4920229004ae84002" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 41123 } }
I20260925 04:48:23.436822 20069 raft_consensus.cc:374] T 2976d2f661c9407691f16a7ee4d6267a P 9dcaef5b60b94a65ae3b0742ac14e43d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:48:23.436929 20069 raft_consensus.cc:729] T 2976d2f661c9407691f16a7ee4d6267a P 9dcaef5b60b94a65ae3b0742ac14e43d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dcaef5b60b94a65ae3b0742ac14e43d, State: Initialized, Role: FOLLOWER
I20260925 04:48:23.437508 20070 ts_tablet_manager.cc:1428] T 2976d2f661c9407691f16a7ee4d6267a P 6429e5ed13e64f4db920b64dc0bc253e: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260925 04:48:23.437474 20069 consensus_queue.cc:260] T 2976d2f661c9407691f16a7ee4d6267a P 9dcaef5b60b94a65ae3b0742ac14e43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcaef5b60b94a65ae3b0742ac14e43d" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43865 } } peers { permanent_uuid: "6429e5ed13e64f4db920b64dc0bc253e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 33545 } } peers { permanent_uuid: "802977ce134243b4920229004ae84002" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 41123 } }
I20260925 04:48:23.438264 19784 heartbeater.cc:503] Master 127.0.98.126:41631 was elected leader, sending a full tablet report...
I20260925 04:48:23.438583 20069 ts_tablet_manager.cc:1428] T 2976d2f661c9407691f16a7ee4d6267a P 9dcaef5b60b94a65ae3b0742ac14e43d: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 04:48:23.440734 20076 raft_consensus.cc:482] T 2976d2f661c9407691f16a7ee4d6267a P 9dcaef5b60b94a65ae3b0742ac14e43d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:48:23.440742 20071 raft_consensus.cc:348] T 2976d2f661c9407691f16a7ee4d6267a P 802977ce134243b4920229004ae84002 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcaef5b60b94a65ae3b0742ac14e43d" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43865 } } peers { permanent_uuid: "6429e5ed13e64f4db920b64dc0bc253e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 33545 } } peers { permanent_uuid: "802977ce134243b4920229004ae84002" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 41123 } }
I20260925 04:48:23.440969 20076 raft_consensus.cc:504] T 2976d2f661c9407691f16a7ee4d6267a P 9dcaef5b60b94a65ae3b0742ac14e43d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcaef5b60b94a65ae3b0742ac14e43d" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43865 } } peers { permanent_uuid: "6429e5ed13e64f4db920b64dc0bc253e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 33545 } } peers { permanent_uuid: "802977ce134243b4920229004ae84002" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 41123 } }
I20260925 04:48:23.441195 20071 raft_consensus.cc:374] T 2976d2f661c9407691f16a7ee4d6267a P 802977ce134243b4920229004ae84002 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:48:23.441313 20071 raft_consensus.cc:729] T 2976d2f661c9407691f16a7ee4d6267a P 802977ce134243b4920229004ae84002 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 802977ce134243b4920229004ae84002, State: Initialized, Role: FOLLOWER
I20260925 04:48:23.441826 20071 consensus_queue.cc:260] T 2976d2f661c9407691f16a7ee4d6267a P 802977ce134243b4920229004ae84002 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcaef5b60b94a65ae3b0742ac14e43d" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43865 } } peers { permanent_uuid: "6429e5ed13e64f4db920b64dc0bc253e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 33545 } } peers { permanent_uuid: "802977ce134243b4920229004ae84002" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 41123 } }
I20260925 04:48:23.442538 20052 heartbeater.cc:503] Master 127.0.98.126:41631 was elected leader, sending a full tablet report...
I20260925 04:48:23.442924 20071 ts_tablet_manager.cc:1428] T 2976d2f661c9407691f16a7ee4d6267a P 802977ce134243b4920229004ae84002: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260925 04:48:23.443783 20076 leader_election.cc:288] T 2976d2f661c9407691f16a7ee4d6267a P 9dcaef5b60b94a65ae3b0742ac14e43d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6429e5ed13e64f4db920b64dc0bc253e (127.0.98.66:33545), 802977ce134243b4920229004ae84002 (127.0.98.67:41123)
I20260925 04:48:23.446060 20077 raft_consensus.cc:482] T 2976d2f661c9407691f16a7ee4d6267a P 802977ce134243b4920229004ae84002 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:48:23.446333 20077 raft_consensus.cc:504] T 2976d2f661c9407691f16a7ee4d6267a P 802977ce134243b4920229004ae84002 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcaef5b60b94a65ae3b0742ac14e43d" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43865 } } peers { permanent_uuid: "6429e5ed13e64f4db920b64dc0bc253e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 33545 } } peers { permanent_uuid: "802977ce134243b4920229004ae84002" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 41123 } }
I20260925 04:48:23.447353 20067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:23.443183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:43641 (local address 127.0.98.66:33545)
0925 04:48:23.443314 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:23.443318 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:23.443332 (+ 14us) server_negotiation.cc:407] Connection header received
0925 04:48:23.443376 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:23.443380 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:23.443427 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:23.443497 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:48:23.443926 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:23.444659 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:23.446187 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:23.446806 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:23.446934 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:23.446999 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:23.447163 (+ 164us) server_negotiation.cc:299] Negotiation successful
0925 04:48:23.447239 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 04:48:23.447358 20078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:23.442610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:33545 (local address 127.0.98.65:43641)
0925 04:48:23.442888 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:23.442905 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 04:48:23.443025 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:48:23.443610 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:48:23.443613 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:48:23.443633 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:48:23.443774 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:23.443781 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:23.444846 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:23.444849 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:23.446059 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:23.446069 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:23.446930 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:23.446934 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:23.447034 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:23.447060 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 04:48:23.447133 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 04:48:23.447199 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":121,"threads_started":1}
I20260925 04:48:23.447958 20079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:23.443570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:41123 (local address 127.0.98.65:58655)
0925 04:48:23.444030 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:23.444045 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:48:23.444133 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:48:23.444463 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:48:23.444466 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:48:23.444482 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:48:23.444592 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:23.444597 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:23.445438 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:23.445442 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:23.446681 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:23.446690 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:23.447548 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:23.447552 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:23.447647 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:23.447669 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 04:48:23.447734 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 04:48:23.447800 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":98,"threads_started":1}
I20260925 04:48:23.447921 19872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2976d2f661c9407691f16a7ee4d6267a" candidate_uuid: "9dcaef5b60b94a65ae3b0742ac14e43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6429e5ed13e64f4db920b64dc0bc253e" is_pre_election: true
I20260925 04:48:23.448208 19872 raft_consensus.cc:2454] T 2976d2f661c9407691f16a7ee4d6267a P 6429e5ed13e64f4db920b64dc0bc253e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dcaef5b60b94a65ae3b0742ac14e43d in term 0.
I20260925 04:48:23.448457 20068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:23.443638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:58655 (local address 127.0.98.67:41123)
0925 04:48:23.443745 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:23.443750 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:23.444133 (+ 383us) server_negotiation.cc:407] Connection header received
0925 04:48:23.444215 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:23.444217 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:23.444268 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:23.444347 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:48:23.444704 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:23.445315 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:23.446794 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:23.447413 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:23.447524 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:23.447577 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:23.448277 (+ 700us) server_negotiation.cc:299] Negotiation successful
0925 04:48:23.448351 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 04:48:23.448992 20006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2976d2f661c9407691f16a7ee4d6267a" candidate_uuid: "9dcaef5b60b94a65ae3b0742ac14e43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "802977ce134243b4920229004ae84002" is_pre_election: true
I20260925 04:48:23.449198 20006 raft_consensus.cc:2454] T 2976d2f661c9407691f16a7ee4d6267a P 802977ce134243b4920229004ae84002 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dcaef5b60b94a65ae3b0742ac14e43d in term 0.
I20260925 04:48:23.449518 19673 leader_election.cc:302] T 2976d2f661c9407691f16a7ee4d6267a P 9dcaef5b60b94a65ae3b0742ac14e43d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6429e5ed13e64f4db920b64dc0bc253e, 9dcaef5b60b94a65ae3b0742ac14e43d; no voters:
I20260925 04:48:23.449774 20076 raft_consensus.cc:2781] T 2976d2f661c9407691f16a7ee4d6267a P 9dcaef5b60b94a65ae3b0742ac14e43d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:48:23.449877 20076 raft_consensus.cc:482] T 2976d2f661c9407691f16a7ee4d6267a P 9dcaef5b60b94a65ae3b0742ac14e43d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:48:23.449932 20076 raft_consensus.cc:3037] T 2976d2f661c9407691f16a7ee4d6267a P 9dcaef5b60b94a65ae3b0742ac14e43d [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:48:23.451211 20077 leader_election.cc:288] T 2976d2f661c9407691f16a7ee4d6267a P 802977ce134243b4920229004ae84002 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dcaef5b60b94a65ae3b0742ac14e43d (127.0.98.65:43865), 6429e5ed13e64f4db920b64dc0bc253e (127.0.98.66:33545)
I20260925 04:48:23.451224 20076 raft_consensus.cc:504] T 2976d2f661c9407691f16a7ee4d6267a P 9dcaef5b60b94a65ae3b0742ac14e43d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcaef5b60b94a65ae3b0742ac14e43d" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43865 } } peers { permanent_uuid: "6429e5ed13e64f4db920b64dc0bc253e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 33545 } } peers { permanent_uuid: "802977ce134243b4920229004ae84002" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 41123 } }
I20260925 04:48:23.451563 20076 leader_election.cc:288] T 2976d2f661c9407691f16a7ee4d6267a P 9dcaef5b60b94a65ae3b0742ac14e43d [CANDIDATE]: Term 1 election: Requested vote from peers 6429e5ed13e64f4db920b64dc0bc253e (127.0.98.66:33545), 802977ce134243b4920229004ae84002 (127.0.98.67:41123)
I20260925 04:48:23.452095 19872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2976d2f661c9407691f16a7ee4d6267a" candidate_uuid: "9dcaef5b60b94a65ae3b0742ac14e43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6429e5ed13e64f4db920b64dc0bc253e"
I20260925 04:48:23.452214 19872 raft_consensus.cc:3037] T 2976d2f661c9407691f16a7ee4d6267a P 6429e5ed13e64f4db920b64dc0bc253e [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:48:23.452347 20067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:23.448196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:41757 (local address 127.0.98.66:33545)
0925 04:48:23.448300 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:23.448304 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:23.448315 (+ 11us) server_negotiation.cc:407] Connection header received
0925 04:48:23.448350 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:23.448353 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:23.448407 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:23.448477 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:48:23.449049 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:23.449710 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:23.451212 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:23.451778 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:23.451907 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:23.451959 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:23.452176 (+ 217us) server_negotiation.cc:299] Negotiation successful
0925 04:48:23.452235 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 04:48:23.452354 20054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:23.447758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:33545 (local address 127.0.98.67:41757)
0925 04:48:23.447979 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:23.447993 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:48:23.448069 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:48:23.448786 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:48:23.448791 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:48:23.448814 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:48:23.448917 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:23.448923 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:23.449838 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:23.449841 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:23.451083 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:23.451093 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:23.451907 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:23.451911 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:23.452043 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:23.452064 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 04:48:23.452162 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 04:48:23.452233 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153}
I20260925 04:48:23.453564 20065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:23.448971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:39943 (local address 127.0.98.65:43865)
0925 04:48:23.449249 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:23.449254 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:23.449358 (+ 104us) server_negotiation.cc:407] Connection header received
0925 04:48:23.449573 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:23.449577 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:23.449620 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:23.449668 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:48:23.450059 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:23.450884 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:23.452365 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:23.452948 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:23.453025 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:23.453077 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:23.453423 (+ 346us) server_negotiation.cc:299] Negotiation successful
0925 04:48:23.453483 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":188}
I20260925 04:48:23.453599 19872 raft_consensus.cc:2454] T 2976d2f661c9407691f16a7ee4d6267a P 6429e5ed13e64f4db920b64dc0bc253e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dcaef5b60b94a65ae3b0742ac14e43d in term 1.
I20260925 04:48:23.453667 20080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:23.448922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:43865 (local address 127.0.98.67:39943)
0925 04:48:23.449261 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:23.449275 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:48:23.449423 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:48:23.449801 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:48:23.449805 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:48:23.449821 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:48:23.449959 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:23.449965 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:23.451032 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:23.451036 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:23.452217 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:23.452229 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:23.453185 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:23.453188 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:23.453293 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:23.453314 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 04:48:23.453398 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 04:48:23.453492 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"mutex_wait_us":3,"thread_start_us":126,"threads_started":1}
I20260925 04:48:23.453902 19673 leader_election.cc:302] T 2976d2f661c9407691f16a7ee4d6267a P 9dcaef5b60b94a65ae3b0742ac14e43d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6429e5ed13e64f4db920b64dc0bc253e, 9dcaef5b60b94a65ae3b0742ac14e43d; no voters:
I20260925 04:48:23.454110 20076 raft_consensus.cc:2781] T 2976d2f661c9407691f16a7ee4d6267a P 9dcaef5b60b94a65ae3b0742ac14e43d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:48:23.454342 19872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2976d2f661c9407691f16a7ee4d6267a" candidate_uuid: "802977ce134243b4920229004ae84002" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6429e5ed13e64f4db920b64dc0bc253e" is_pre_election: true
I20260925 04:48:23.454366 20076 raft_consensus.cc:686] T 2976d2f661c9407691f16a7ee4d6267a P 9dcaef5b60b94a65ae3b0742ac14e43d [term 1 LEADER]: Becoming Leader. State: Replica: 9dcaef5b60b94a65ae3b0742ac14e43d, State: Running, Role: LEADER
I20260925 04:48:23.454345 19738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2976d2f661c9407691f16a7ee4d6267a" candidate_uuid: "802977ce134243b4920229004ae84002" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dcaef5b60b94a65ae3b0742ac14e43d" is_pre_election: true
I20260925 04:48:23.454523 19872 raft_consensus.cc:2379] T 2976d2f661c9407691f16a7ee4d6267a P 6429e5ed13e64f4db920b64dc0bc253e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 802977ce134243b4920229004ae84002 in current term 1: Already voted for candidate 9dcaef5b60b94a65ae3b0742ac14e43d in this term.
I20260925 04:48:23.454756 20076 consensus_queue.cc:237] T 2976d2f661c9407691f16a7ee4d6267a P 9dcaef5b60b94a65ae3b0742ac14e43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcaef5b60b94a65ae3b0742ac14e43d" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43865 } } peers { permanent_uuid: "6429e5ed13e64f4db920b64dc0bc253e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 33545 } } peers { permanent_uuid: "802977ce134243b4920229004ae84002" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 41123 } }
I20260925 04:48:23.455107 19942 leader_election.cc:302] T 2976d2f661c9407691f16a7ee4d6267a P 802977ce134243b4920229004ae84002 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 802977ce134243b4920229004ae84002; no voters: 6429e5ed13e64f4db920b64dc0bc253e, 9dcaef5b60b94a65ae3b0742ac14e43d
I20260925 04:48:23.455404 20077 raft_consensus.cc:3037] T 2976d2f661c9407691f16a7ee4d6267a P 802977ce134243b4920229004ae84002 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:48:23.455637 20006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2976d2f661c9407691f16a7ee4d6267a" candidate_uuid: "9dcaef5b60b94a65ae3b0742ac14e43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "802977ce134243b4920229004ae84002"
I20260925 04:48:23.456670 20077 raft_consensus.cc:2726] T 2976d2f661c9407691f16a7ee4d6267a P 802977ce134243b4920229004ae84002 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 04:48:23.457832 20006 raft_consensus.cc:2454] T 2976d2f661c9407691f16a7ee4d6267a P 802977ce134243b4920229004ae84002 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dcaef5b60b94a65ae3b0742ac14e43d in term 1.
I20260925 04:48:23.458551 19592 catalog_manager.cc:5495] T 2976d2f661c9407691f16a7ee4d6267a P 9dcaef5b60b94a65ae3b0742ac14e43d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9dcaef5b60b94a65ae3b0742ac14e43d (127.0.98.65). New cstate: current_term: 1 leader_uuid: "9dcaef5b60b94a65ae3b0742ac14e43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcaef5b60b94a65ae3b0742ac14e43d" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6429e5ed13e64f4db920b64dc0bc253e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 33545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "802977ce134243b4920229004ae84002" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 41123 } health_report { overall_health: UNKNOWN } } }
I20260925 04:48:23.479758 20065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:23.476312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38068 (local address 127.0.98.65:43865)
0925 04:48:23.476436 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:23.476440 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:23.476452 (+ 12us) server_negotiation.cc:407] Connection header received
0925 04:48:23.476544 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:23.476546 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:23.476598 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:23.476685 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:48:23.476928 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:23.477436 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:23.478396 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:23.478608 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:23.478685 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:23.478973 (+ 288us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:48:23.479388 (+ 415us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:48:23.479478 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:23.479596 (+ 118us) server_negotiation.cc:299] Negotiation successful
0925 04:48:23.479646 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 04:48:23.485846 19872 raft_consensus.cc:1261] T 2976d2f661c9407691f16a7ee4d6267a P 6429e5ed13e64f4db920b64dc0bc253e [term 1 FOLLOWER]: Refusing update from remote peer 9dcaef5b60b94a65ae3b0742ac14e43d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:48:23.485888 20006 raft_consensus.cc:1261] T 2976d2f661c9407691f16a7ee4d6267a P 802977ce134243b4920229004ae84002 [term 1 FOLLOWER]: Refusing update from remote peer 9dcaef5b60b94a65ae3b0742ac14e43d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:48:23.486603 20081 consensus_queue.cc:1035] T 2976d2f661c9407691f16a7ee4d6267a P 9dcaef5b60b94a65ae3b0742ac14e43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "802977ce134243b4920229004ae84002" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 41123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:48:23.486819 20076 consensus_queue.cc:1035] T 2976d2f661c9407691f16a7ee4d6267a P 9dcaef5b60b94a65ae3b0742ac14e43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6429e5ed13e64f4db920b64dc0bc253e" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 33545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:48:23.495253 20084 mvcc.cc:205] Tried to move back new op lower bound from 7333116737470517248 to 7333116737354031104. Current Snapshot: MvccSnapshot[applied={T|T < 7333116737470517248 or (T in {7333116737470517248})}]
I20260925 04:48:23.497908 20086 mvcc.cc:205] Tried to move back new op lower bound from 7333116737470517248 to 7333116737354031104. Current Snapshot: MvccSnapshot[applied={T|T < 7333116737470517248}]
I20260925 04:48:23.503091 19592 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36348:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:48:23.503280 19592 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36348:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:48:23.505725 19592 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 842e16d48f974f1da0c0d3c40548fdc4: Sending DeleteTablet for 3 replicas of tablet 2976d2f661c9407691f16a7ee4d6267a
I20260925 04:48:23.506549 19718 tablet_service.cc:1514] Processing DeleteTablet for tablet 2976d2f661c9407691f16a7ee4d6267a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:48:23 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:38056
I20260925 04:48:23.506915 19852 tablet_service.cc:1514] Processing DeleteTablet for tablet 2976d2f661c9407691f16a7ee4d6267a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:48:23 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:45846
I20260925 04:48:23.507319 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 19655
I20260925 04:48:23.507432 19986 tablet_service.cc:1514] Processing DeleteTablet for tablet 2976d2f661c9407691f16a7ee4d6267a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:48:23 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:34980
I20260925 04:48:23.508231 20095 tablet_replica.cc:330] T 2976d2f661c9407691f16a7ee4d6267a P 6429e5ed13e64f4db920b64dc0bc253e: stopping tablet replica
I20260925 04:48:23.508610 20095 raft_consensus.cc:2229] T 2976d2f661c9407691f16a7ee4d6267a P 6429e5ed13e64f4db920b64dc0bc253e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:48:23.508970 20095 raft_consensus.cc:2258] T 2976d2f661c9407691f16a7ee4d6267a P 6429e5ed13e64f4db920b64dc0bc253e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:48:23.510125 20096 tablet_replica.cc:330] T 2976d2f661c9407691f16a7ee4d6267a P 802977ce134243b4920229004ae84002: stopping tablet replica
I20260925 04:48:23.510493 20096 raft_consensus.cc:2229] T 2976d2f661c9407691f16a7ee4d6267a P 802977ce134243b4920229004ae84002 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:48:23.510522 20095 ts_tablet_manager.cc:1905] T 2976d2f661c9407691f16a7ee4d6267a P 6429e5ed13e64f4db920b64dc0bc253e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:48:23.510823 20096 raft_consensus.cc:2258] T 2976d2f661c9407691f16a7ee4d6267a P 802977ce134243b4920229004ae84002 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:48:23.512364 20096 ts_tablet_manager.cc:1905] T 2976d2f661c9407691f16a7ee4d6267a P 802977ce134243b4920229004ae84002: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:48:23.513446 20095 ts_tablet_manager.cc:1918] T 2976d2f661c9407691f16a7ee4d6267a P 6429e5ed13e64f4db920b64dc0bc253e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:48:23.513552 20095 log.cc:1196] T 2976d2f661c9407691f16a7ee4d6267a P 6429e5ed13e64f4db920b64dc0bc253e: Deleting WAL directory at /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/minicluster-data/ts-1/wal/wals/2976d2f661c9407691f16a7ee4d6267a
I20260925 04:48:23.513972 20095 ts_tablet_manager.cc:1939] T 2976d2f661c9407691f16a7ee4d6267a P 6429e5ed13e64f4db920b64dc0bc253e: Deleting consensus metadata
I20260925 04:48:23.515017 20096 ts_tablet_manager.cc:1918] T 2976d2f661c9407691f16a7ee4d6267a P 802977ce134243b4920229004ae84002: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:48:23.515147 20096 log.cc:1196] T 2976d2f661c9407691f16a7ee4d6267a P 802977ce134243b4920229004ae84002: Deleting WAL directory at /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790311449873301-393-0/minicluster-data/ts-2/wal/wals/2976d2f661c9407691f16a7ee4d6267a
I20260925 04:48:23.515511 20096 ts_tablet_manager.cc:1939] T 2976d2f661c9407691f16a7ee4d6267a P 802977ce134243b4920229004ae84002: Deleting consensus metadata
I20260925 04:48:23.515486 19579 catalog_manager.cc:4841] TS 6429e5ed13e64f4db920b64dc0bc253e (127.0.98.66:33545): tablet 2976d2f661c9407691f16a7ee4d6267a (table test-table [id=cbfb30b358a34d27a8ba73d990452c5f]) successfully deleted
W20260925 04:48:23.515488 19580 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.65:43865 (error 108)
I20260925 04:48:23.515697 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 19789
I20260925 04:48:23.516686 19579 catalog_manager.cc:4841] TS 802977ce134243b4920229004ae84002 (127.0.98.67:41123): tablet 2976d2f661c9407691f16a7ee4d6267a (table test-table [id=cbfb30b358a34d27a8ba73d990452c5f]) successfully deleted
I20260925 04:48:23.517261 20066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:23.516923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:43865 (local address 127.0.0.1:38070)
0925 04:48:23.517069 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:23.517152 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.65:43865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 04:48:23.517432 19580 catalog_manager.cc:4584] TS 9dcaef5b60b94a65ae3b0742ac14e43d (127.0.98.65:43865): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2976d2f661c9407691f16a7ee4d6267a: Network error: Client connection negotiation failed: client connection to 127.0.98.65:43865: connect: Connection refused (error 111)
I20260925 04:48:23.522660 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 19923
I20260925 04:48:23.528365 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 19561
2026-09-25T04:48:23Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2249 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-25T04:48:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:48:23Z Disabled control of system clock
I20260925 04:48:23.559347 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311449873301-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311449873301-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311449873301-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:37957
--webserver_interface=127.0.98.126
--webserver_port=0
--builtin_ntp_servers=127.0.98.84:42199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:37957
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 04:48:23.669708 20101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:48:23.670007 20101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:48:23.670044 20101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:48:23.674019 20101 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:48:23.674084 20101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:48:23.674108 20101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:48:23.674139 20101 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:48:23.674154 20101 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:48:23.679028 20101 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:42199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311449873301-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311449873301-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:37957
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:37957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20101
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311449873301-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:48:23.680035 20101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:48:23.681365 20101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:48:23.689803 20106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:48:23.689851 20109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:48:23.689890 20107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:48:23.690167 20101 server_base.cc:1034] running on GCE node
I20260925 04:48:23.690747 20101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:48:23.691799 20101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:48:23.692963 20101 hybrid_clock.cc:648] HybridClock initialized: now 1790311703692936 us; error 48 us; skew 500 ppm
I20260925 04:48:23.694958 20101 webserver.cc:460] Webserver started at http://127.0.98.126:41207/ using document root <none> and password file <none>
I20260925 04:48:23.695538 20101 fs_manager.cc:359] Metadata directory not provided
I20260925 04:48:23.695603 20101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:48:23.695822 20101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:48:23.697918 20101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311449873301-393-0/minicluster-data/master-0/data/instance:
uuid: "4b81bff3f581432b9157c60a631dcc92"
format_stamp: "Formatted at 2026-09-25 04:48:23 on dist-test-slave-kghz"
server_key: "2189d53c022f46a40eaad0063fe53f1d"
server_key_iv: "afdf6daccefeb37c6b697a64feaaf379"
server_key_version: "encryptionkey@0"
I20260925 04:48:23.698539 20101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311449873301-393-0/minicluster-data/master-0/wal/instance:
uuid: "4b81bff3f581432b9157c60a631dcc92"
format_stamp: "Formatted at 2026-09-25 04:48:23 on dist-test-slave-kghz"
server_key: "2189d53c022f46a40eaad0063fe53f1d"
server_key_iv: "afdf6daccefeb37c6b697a64feaaf379"
server_key_version: "encryptionkey@0"
I20260925 04:48:23.702343 20101 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260925 04:48:23.704932 20115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:48:23.706427 20101 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260925 04:48:23.706586 20101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311449873301-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311449873301-393-0/minicluster-data/master-0/wal
uuid: "4b81bff3f581432b9157c60a631dcc92"
format_stamp: "Formatted at 2026-09-25 04:48:23 on dist-test-slave-kghz"
server_key: "2189d53c022f46a40eaad0063fe53f1d"
server_key_iv: "afdf6daccefeb37c6b697a64feaaf379"
server_key_version: "encryptionkey@0"
I20260925 04:48:23.706720 20101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311449873301-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311449873301-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311449873301-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:48:23.725769 20101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:48:23.726586 20101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:48:23.726809 20101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:48:23.734284 20167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:37957 every 8 connection(s)
I20260925 04:48:23.734285 20101 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:37957
I20260925 04:48:23.735303 20101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
I20260925 04:48:23.738457 20168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:48:23.743988 20168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b81bff3f581432b9157c60a631dcc92: Bootstrap starting.
I20260925 04:48:23.745224 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 20101
I20260925 04:48:23.745376 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311449873301-393-0/minicluster-data/master-0/wal/instance
I20260925 04:48:23.745687 393 external_mini_cluster.cc:1428] Setting key 0ba3ff1628056c8e2480fa2c15cf1537
I20260925 04:48:23.747098 20168 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b81bff3f581432b9157c60a631dcc92: Neither blocks nor log segments found. Creating new log.
I20260925 04:48:23.747924 20168 log.cc:824] T 00000000000000000000000000000000 P 4b81bff3f581432b9157c60a631dcc92: Log is configured to *not* fsync() on all Append() calls
I20260925 04:48:23.750496 20168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b81bff3f581432b9157c60a631dcc92: No bootstrap required, opened a new log
I20260925 04:48:23.754944 20168 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b81bff3f581432b9157c60a631dcc92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b81bff3f581432b9157c60a631dcc92" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 37957 } }
I20260925 04:48:23.755384 20168 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b81bff3f581432b9157c60a631dcc92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:48:23.755461 20168 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b81bff3f581432b9157c60a631dcc92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b81bff3f581432b9157c60a631dcc92, State: Initialized, Role: FOLLOWER
I20260925 04:48:23.755970 20168 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b81bff3f581432b9157c60a631dcc92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b81bff3f581432b9157c60a631dcc92" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 37957 } }
I20260925 04:48:23.756129 20168 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b81bff3f581432b9157c60a631dcc92 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:48:23.756217 20168 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b81bff3f581432b9157c60a631dcc92 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:48:23.756323 20168 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b81bff3f581432b9157c60a631dcc92 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:48:23.756446 20171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:23.747087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47850 (local address 127.0.98.126:37957)
0925 04:48:23.747834 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:23.747849 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:23.747886 (+ 37us) server_negotiation.cc:407] Connection header received
0925 04:48:23.748833 (+ 947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:23.748863 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:23.749250 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:23.749578 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:23.750133 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:23.751337 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:23.752633 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:23.753149 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:23.753272 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:23.754373 (+ 1101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:23.754378 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:23.754393 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:48:23.754838 (+ 445us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:23.754900 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:48:23.754922 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:23.755008 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:23.755180 (+ 172us) server_negotiation.cc:299] Negotiation successful
0925 04:48:23.755512 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":171,"threads_started":1}
I20260925 04:48:23.757597 20168 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b81bff3f581432b9157c60a631dcc92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b81bff3f581432b9157c60a631dcc92" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 37957 } }
I20260925 04:48:23.757982 20168 leader_election.cc:302] T 00000000000000000000000000000000 P 4b81bff3f581432b9157c60a631dcc92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b81bff3f581432b9157c60a631dcc92; no voters:
I20260925 04:48:23.758368 20168 leader_election.cc:288] T 00000000000000000000000000000000 P 4b81bff3f581432b9157c60a631dcc92 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:48:23.758456 20173 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b81bff3f581432b9157c60a631dcc92 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:48:23.758670 20173 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b81bff3f581432b9157c60a631dcc92 [term 1 LEADER]: Becoming Leader. State: Replica: 4b81bff3f581432b9157c60a631dcc92, State: Running, Role: LEADER
I20260925 04:48:23.759068 20173 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b81bff3f581432b9157c60a631dcc92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b81bff3f581432b9157c60a631dcc92" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 37957 } }
I20260925 04:48:23.759653 20168 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b81bff3f581432b9157c60a631dcc92 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:48:23.760221 20175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b81bff3f581432b9157c60a631dcc92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b81bff3f581432b9157c60a631dcc92. Latest consensus state: current_term: 1 leader_uuid: "4b81bff3f581432b9157c60a631dcc92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b81bff3f581432b9157c60a631dcc92" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 37957 } } }
I20260925 04:48:23.760455 20175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b81bff3f581432b9157c60a631dcc92 [sys.catalog]: This master's current role is: LEADER
I20260925 04:48:23.760634 20174 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b81bff3f581432b9157c60a631dcc92 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b81bff3f581432b9157c60a631dcc92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b81bff3f581432b9157c60a631dcc92" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 37957 } } }
I20260925 04:48:23.760725 20174 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b81bff3f581432b9157c60a631dcc92 [sys.catalog]: This master's current role is: LEADER
I20260925 04:48:23.760922 20181 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:48:23.764138 20181 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:48:23.769459 20181 catalog_manager.cc:1347] Generated new cluster ID: 4a58187d847940419a9a3df9c1adf3e8
I20260925 04:48:23.769543 20181 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:48:23.786594 20181 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:48:23.787675 20181 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:48:23.805307 20181 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b81bff3f581432b9157c60a631dcc92: Generated new TSK 0
I20260925 04:48:23.806084 20181 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:48:23.812057 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311449873301-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311449873301-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311449873301-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:0
--local_ip_for_outbound_sockets=127.0.98.65
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:37957
--builtin_ntp_servers=127.0.98.84:42199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 04:48:23.942819 20192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:48:23.943064 20192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:48:23.943102 20192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:48:23.947110 20192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:48:23.947165 20192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:48:23.947225 20192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260925 04:48:23.952054 20192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:42199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311449873301-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311449873301-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:37957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311449873301-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:48:23.953150 20192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:48:23.954618 20192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:48:23.964676 20197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:48:23.964875 20192 server_base.cc:1034] running on GCE node
W20260925 04:48:23.964672 20200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:48:23.964668 20198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:48:23.965471 20192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:48:23.966112 20192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:48:23.967298 20192 hybrid_clock.cc:648] HybridClock initialized: now 1790311703967267 us; error 52 us; skew 500 ppm
I20260925 04:48:23.969105 20192 webserver.cc:460] Webserver started at http://127.0.98.65:44699/ using document root <none> and password file <none>
I20260925 04:48:23.969689 20192 fs_manager.cc:359] Metadata directory not provided
I20260925 04:48:23.969785 20192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:48:23.970019 20192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:48:23.972142 20192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311449873301-393-0/minicluster-data/ts-0/data/instance:
uuid: "50a302a85e9942bca33a9de7abe5ff3a"
format_stamp: "Formatted at 2026-09-25 04:48:23 on dist-test-slave-kghz"
server_key: "65a2dea6906ff079a8e2ecb7f2826fd1"
server_key_iv: "f2f9d3a727f8b94bc934e17dc51a5192"
server_key_version: "encryptionkey@0"
I20260925 04:48:23.972738 20192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311449873301-393-0/minicluster-data/ts-0/wal/instance:
uuid: "50a302a85e9942bca33a9de7abe5ff3a"
format_stamp: "Formatted at 2026-09-25 04:48:23 on dist-test-slave-kghz"
server_key: "65a2dea6906ff079a8e2ecb7f2826fd1"
server_key_iv: "f2f9d3a727f8b94bc934e17dc51a5192"
server_key_version: "encryptionkey@0"
I20260925 04:48:23.976466 20192 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 04:48:23.979087 20206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:48:23.980357 20192 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 04:48:23.980490 20192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311449873301-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311449873301-393-0/minicluster-data/ts-0/wal
uuid: "50a302a85e9942bca33a9de7abe5ff3a"
format_stamp: "Formatted at 2026-09-25 04:48:23 on dist-test-slave-kghz"
server_key: "65a2dea6906ff079a8e2ecb7f2826fd1"
server_key_iv: "f2f9d3a727f8b94bc934e17dc51a5192"
server_key_version: "encryptionkey@0"
I20260925 04:48:23.980602 20192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311449873301-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311449873301-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311449873301-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:48:24.000219 20192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:48:24.000946 20192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:48:24.001150 20192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:48:24.001756 20192 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:48:24.002779 20192 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:48:24.002854 20192 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:48:24.002924 20192 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:48:24.002974 20192 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:48:24.012148 20192 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:41765
I20260925 04:48:24.012187 20319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:41765 every 8 connection(s)
I20260925 04:48:24.013093 20192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311449873301-393-0/minicluster-data/ts-0/data/info.pb
I20260925 04:48:24.018695 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 20192
I20260925 04:48:24.018826 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311449873301-393-0/minicluster-data/ts-0/wal/instance
I20260925 04:48:24.019109 393 external_mini_cluster.cc:1428] Setting key 4f88f48cba45da5382c8c69dd8a845fb
I20260925 04:48:24.021258 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311449873301-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311449873301-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311449873301-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:0
--local_ip_for_outbound_sockets=127.0.98.66
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:37957
--builtin_ntp_servers=127.0.98.84:42199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 04:48:24.023511 20171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:24.015081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:57373 (local address 127.0.98.126:37957)
0925 04:48:24.015228 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:24.015232 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:24.016227 (+ 995us) server_negotiation.cc:407] Connection header received
0925 04:48:24.017575 (+ 1348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:24.017578 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:24.017626 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:24.017715 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:24.018847 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:24.019613 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:24.021185 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:24.021494 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:24.021632 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:24.022966 (+ 1334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:24.022971 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:24.022975 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:48:24.022991 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:24.023016 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:48:24.023029 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:24.023116 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:24.023258 (+ 142us) server_negotiation.cc:299] Negotiation successful
0925 04:48:24.023330 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 04:48:24.024327 20322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:24.015438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:37957 (local address 127.0.98.65:57373)
0925 04:48:24.016084 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:24.016125 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 04:48:24.017382 (+ 1257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:48:24.017897 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:48:24.017903 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:48:24.017986 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:48:24.018636 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:24.018657 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:24.019815 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:24.019819 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:24.021052 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:24.021067 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:24.021789 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:24.021792 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:24.021943 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:24.022651 (+ 708us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:48:24.022653 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:48:24.022712 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:48:24.022782 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:48:24.023121 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:48:24.023128 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:48:24.023132 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 04:48:24.023227 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 04:48:24.023448 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":124,"threads_started":1}
I20260925 04:48:24.025575 20320 heartbeater.cc:347] Connected to a master server at 127.0.98.126:37957
I20260925 04:48:24.025851 20320 heartbeater.cc:464] Registering TS with master...
I20260925 04:48:24.026435 20320 heartbeater.cc:511] Master 127.0.98.126:37957 requested a full tablet report, sending...
I20260925 04:48:24.028268 20130 ts_manager.cc:194] Registered new tserver with Master: 50a302a85e9942bca33a9de7abe5ff3a (127.0.98.65:41765)
I20260925 04:48:24.030035 20130 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.65:57373
W20260925 04:48:24.132802 20323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:48:24.133080 20323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:48:24.133117 20323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:48:24.137060 20323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:48:24.137120 20323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:48:24.137177 20323 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260925 04:48:24.141906 20323 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:42199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311449873301-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311449873301-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:37957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20323
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311449873301-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:48:24.143020 20323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:48:24.144397 20323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:48:24.154139 20331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:48:24.154381 20323 server_base.cc:1034] running on GCE node
W20260925 04:48:24.154158 20328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:48:24.154139 20329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:48:24.154875 20323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:48:24.155542 20323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:48:24.156729 20323 hybrid_clock.cc:648] HybridClock initialized: now 1790311704156712 us; error 29 us; skew 500 ppm
I20260925 04:48:24.158584 20323 webserver.cc:460] Webserver started at http://127.0.98.66:43599/ using document root <none> and password file <none>
I20260925 04:48:24.159201 20323 fs_manager.cc:359] Metadata directory not provided
I20260925 04:48:24.159293 20323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:48:24.159515 20323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:48:24.161784 20323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311449873301-393-0/minicluster-data/ts-1/data/instance:
uuid: "d4e70b1e2be640f9ac5f4d567d70bfa9"
format_stamp: "Formatted at 2026-09-25 04:48:24 on dist-test-slave-kghz"
server_key: "95a57aa24a4ba4324dace678bb786ceb"
server_key_iv: "b1510aef042fc01b6a2ef2aef03f40e2"
server_key_version: "encryptionkey@0"
I20260925 04:48:24.162398 20323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311449873301-393-0/minicluster-data/ts-1/wal/instance:
uuid: "d4e70b1e2be640f9ac5f4d567d70bfa9"
format_stamp: "Formatted at 2026-09-25 04:48:24 on dist-test-slave-kghz"
server_key: "95a57aa24a4ba4324dace678bb786ceb"
server_key_iv: "b1510aef042fc01b6a2ef2aef03f40e2"
server_key_version: "encryptionkey@0"
I20260925 04:48:24.165937 20323 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:48:24.168381 20337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:48:24.169612 20323 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 04:48:24.169756 20323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311449873301-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311449873301-393-0/minicluster-data/ts-1/wal
uuid: "d4e70b1e2be640f9ac5f4d567d70bfa9"
format_stamp: "Formatted at 2026-09-25 04:48:24 on dist-test-slave-kghz"
server_key: "95a57aa24a4ba4324dace678bb786ceb"
server_key_iv: "b1510aef042fc01b6a2ef2aef03f40e2"
server_key_version: "encryptionkey@0"
I20260925 04:48:24.169870 20323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311449873301-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311449873301-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311449873301-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:48:24.201519 20323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:48:24.202342 20323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:48:24.202574 20323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:48:24.203199 20323 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:48:24.204186 20323 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:48:24.204268 20323 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:48:24.204336 20323 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:48:24.204380 20323 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:48:24.213369 20323 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:34437
I20260925 04:48:24.213450 20450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:34437 every 8 connection(s)
I20260925 04:48:24.214457 20323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311449873301-393-0/minicluster-data/ts-1/data/info.pb
I20260925 04:48:24.218348 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 20323
I20260925 04:48:24.218489 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311449873301-393-0/minicluster-data/ts-1/wal/instance
I20260925 04:48:24.218881 393 external_mini_cluster.cc:1428] Setting key bf8f508860618e186786cc52915246c1
I20260925 04:48:24.221026 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311449873301-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311449873301-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311449873301-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:0
--local_ip_for_outbound_sockets=127.0.98.67
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:37957
--builtin_ntp_servers=127.0.98.84:42199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 04:48:24.225311 20171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:24.216280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:53537 (local address 127.0.98.126:37957)
0925 04:48:24.216429 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:24.216433 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:24.217477 (+ 1044us) server_negotiation.cc:407] Connection header received
0925 04:48:24.219342 (+ 1865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:24.219348 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:24.219413 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:24.219507 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:24.220642 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:24.221246 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:24.222969 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:24.223357 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:24.223489 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:24.224630 (+ 1141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:24.224637 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:24.224643 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:48:24.224675 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:24.224703 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:48:24.224717 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:24.224802 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:24.225004 (+ 202us) server_negotiation.cc:299] Negotiation successful
0925 04:48:24.225086 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 04:48:24.226096 20453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:24.216648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:37957 (local address 127.0.98.66:53537)
0925 04:48:24.217324 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:24.217366 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 04:48:24.219076 (+ 1710us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:48:24.219713 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:48:24.219722 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:48:24.219829 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:48:24.220484 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:24.220505 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:24.221393 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:24.221401 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:24.222841 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:24.222851 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:24.223522 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:24.223525 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:24.223673 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:24.224339 (+ 666us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:48:24.224340 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:48:24.224389 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:48:24.224460 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:48:24.224820 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:48:24.224833 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:48:24.224839 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 04:48:24.224973 (+ 134us) client_negotiation.cc:240] Negotiation successful
0925 04:48:24.225214 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":112,"threads_started":1}
I20260925 04:48:24.227404 20451 heartbeater.cc:347] Connected to a master server at 127.0.98.126:37957
I20260925 04:48:24.227643 20451 heartbeater.cc:464] Registering TS with master...
I20260925 04:48:24.228188 20451 heartbeater.cc:511] Master 127.0.98.126:37957 requested a full tablet report, sending...
I20260925 04:48:24.229321 20130 ts_manager.cc:194] Registered new tserver with Master: d4e70b1e2be640f9ac5f4d567d70bfa9 (127.0.98.66:34437)
I20260925 04:48:24.229929 20130 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.66:53537
W20260925 04:48:24.335706 20454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:48:24.335973 20454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:48:24.336012 20454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:48:24.340049 20454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:48:24.340106 20454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:48:24.340161 20454 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260925 04:48:24.344926 20454 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:42199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311449873301-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311449873301-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:37957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311449873301-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:48:24.346103 20454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:48:24.347561 20454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:48:24.357208 20460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:48:24.357244 20459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:48:24.357378 20454 server_base.cc:1034] running on GCE node
W20260925 04:48:24.357244 20462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:48:24.357913 20454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:48:24.358613 20454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:48:24.359772 20454 hybrid_clock.cc:648] HybridClock initialized: now 1790311704359758 us; error 25 us; skew 500 ppm
I20260925 04:48:24.361919 20454 webserver.cc:460] Webserver started at http://127.0.98.67:43565/ using document root <none> and password file <none>
I20260925 04:48:24.362552 20454 fs_manager.cc:359] Metadata directory not provided
I20260925 04:48:24.362617 20454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:48:24.362875 20454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:48:24.364905 20454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311449873301-393-0/minicluster-data/ts-2/data/instance:
uuid: "359a9cda7084405a9a9bc572f991145b"
format_stamp: "Formatted at 2026-09-25 04:48:24 on dist-test-slave-kghz"
server_key: "17a2cd838cb98bab05437cde60b72cc2"
server_key_iv: "154bfbdb765e14347115aa2ac808b76a"
server_key_version: "encryptionkey@0"
I20260925 04:48:24.365480 20454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311449873301-393-0/minicluster-data/ts-2/wal/instance:
uuid: "359a9cda7084405a9a9bc572f991145b"
format_stamp: "Formatted at 2026-09-25 04:48:24 on dist-test-slave-kghz"
server_key: "17a2cd838cb98bab05437cde60b72cc2"
server_key_iv: "154bfbdb765e14347115aa2ac808b76a"
server_key_version: "encryptionkey@0"
I20260925 04:48:24.369506 20454 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 04:48:24.372182 20468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:48:24.373616 20454 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 04:48:24.373714 20454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311449873301-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311449873301-393-0/minicluster-data/ts-2/wal
uuid: "359a9cda7084405a9a9bc572f991145b"
format_stamp: "Formatted at 2026-09-25 04:48:24 on dist-test-slave-kghz"
server_key: "17a2cd838cb98bab05437cde60b72cc2"
server_key_iv: "154bfbdb765e14347115aa2ac808b76a"
server_key_version: "encryptionkey@0"
I20260925 04:48:24.373855 20454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311449873301-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311449873301-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311449873301-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:48:24.392818 20454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:48:24.393494 20454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:48:24.393646 20454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:48:24.394178 20454 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:48:24.395157 20454 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:48:24.395205 20454 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:48:24.395243 20454 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:48:24.395259 20454 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:48:24.403858 20454 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:39455
I20260925 04:48:24.403925 20581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:39455 every 8 connection(s)
I20260925 04:48:24.404829 20454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311449873301-393-0/minicluster-data/ts-2/data/info.pb
I20260925 04:48:24.407815 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 20454
I20260925 04:48:24.407943 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311449873301-393-0/minicluster-data/ts-2/wal/instance
I20260925 04:48:24.408232 393 external_mini_cluster.cc:1428] Setting key 3d88e7a9a693a1812f6956f44a9d06e8
I20260925 04:48:24.414650 20171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:24.406783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:51065 (local address 127.0.98.126:37957)
0925 04:48:24.406945 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:24.406949 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:24.408002 (+ 1053us) server_negotiation.cc:407] Connection header received
0925 04:48:24.409678 (+ 1676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:24.409681 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:24.409748 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:24.409846 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:24.410893 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:24.411518 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:24.412669 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:24.412899 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:24.413015 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:24.414115 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:24.414123 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:24.414126 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:48:24.414138 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:24.414157 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:48:24.414165 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:24.414247 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:24.414446 (+ 199us) server_negotiation.cc:299] Negotiation successful
0925 04:48:24.414511 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 04:48:24.415493 20584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:24.407135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:37957 (local address 127.0.98.67:51065)
0925 04:48:24.407827 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:24.407868 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 04:48:24.409473 (+ 1605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:48:24.410025 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:48:24.410032 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:48:24.410160 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:48:24.410753 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:24.410767 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:24.411643 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:24.411647 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:24.412538 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:24.412547 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:24.413054 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:24.413058 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:24.413236 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:24.413830 (+ 594us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:48:24.413832 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:48:24.413872 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:48:24.413937 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:48:24.414288 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:48:24.414297 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:48:24.414303 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 04:48:24.414444 (+ 141us) client_negotiation.cc:240] Negotiation successful
0925 04:48:24.414721 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":114,"threads_started":1}
I20260925 04:48:24.416550 20582 heartbeater.cc:347] Connected to a master server at 127.0.98.126:37957
I20260925 04:48:24.416760 20582 heartbeater.cc:464] Registering TS with master...
I20260925 04:48:24.417214 20582 heartbeater.cc:511] Master 127.0.98.126:37957 requested a full tablet report, sending...
I20260925 04:48:24.418184 20130 ts_manager.cc:194] Registered new tserver with Master: 359a9cda7084405a9a9bc572f991145b (127.0.98.67:39455)
I20260925 04:48:24.418802 20130 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.67:51065
I20260925 04:48:24.422027 393 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 04:48:24.427882 20171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:24.424117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47864 (local address 127.0.98.126:37957)
0925 04:48:24.424240 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:24.424243 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:24.424415 (+ 172us) server_negotiation.cc:407] Connection header received
0925 04:48:24.424513 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:24.424515 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:24.424552 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:24.424620 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:24.424951 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:24.425447 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:24.426375 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:24.426590 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:24.426674 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:24.427464 (+ 790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:24.427467 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:24.427469 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:48:24.427481 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:24.427498 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:48:24.427505 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:24.427565 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:24.427720 (+ 155us) server_negotiation.cc:299] Negotiation successful
0925 04:48:24.427770 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 04:48:24.430783 20130 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47864:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:48:24.434404 20130 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:48:24.449433 20594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:24.443039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:41765 (local address 127.0.0.1:59662)
0925 04:48:24.443686 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:24.443741 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 04:48:24.443875 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:48:24.445074 (+ 1199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:48:24.445077 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:48:24.445108 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:48:24.445286 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:24.445294 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:24.446527 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:24.446532 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:24.447886 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:24.447895 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:24.448831 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:24.448835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:24.448970 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:24.449025 (+ 55us) client_negotiation.cc:769] Sending connection context
0925 04:48:24.449131 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 04:48:24.449258 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"mutex_wait_us":238,"spinlock_wait_cycles":7040,"thread_start_us":83,"threads_started":1}
I20260925 04:48:24.449452 20595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:24.443323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59662 (local address 127.0.98.65:41765)
0925 04:48:24.444656 (+ 1333us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:24.444672 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:24.444698 (+ 26us) server_negotiation.cc:407] Connection header received
0925 04:48:24.444777 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:24.444781 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:24.444917 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:24.445196 (+ 279us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:48:24.445419 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:24.446419 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:24.448005 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:24.448694 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:24.448808 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:24.448996 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:24.449182 (+ 186us) server_negotiation.cc:299] Negotiation successful
0925 04:48:24.449286 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1217,"thread_start_us":135,"threads_started":1}
I20260925 04:48:24.449842 20597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:24.443828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44954 (local address 127.0.98.67:39455)
0925 04:48:24.444431 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:24.444444 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:24.444592 (+ 148us) server_negotiation.cc:407] Connection header received
0925 04:48:24.444680 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:24.444686 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:24.444893 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:24.445053 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:48:24.445602 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:24.446752 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:24.448315 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:24.448903 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:24.449344 (+ 441us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:24.449524 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:24.449578 (+ 54us) server_negotiation.cc:299] Negotiation successful
0925 04:48:24.449690 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":141,"threads_started":1}
I20260925 04:48:24.449883 20596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:24.443796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:34437 (local address 127.0.0.1:46524)
0925 04:48:24.444291 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:24.444304 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:48:24.444377 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:48:24.445600 (+ 1223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:48:24.445603 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:48:24.445618 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:48:24.445724 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:24.445730 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:24.447093 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:24.447097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:24.448231 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:24.448244 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:24.449481 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:24.449485 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:24.449585 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:24.449608 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 04:48:24.449680 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 04:48:24.449752 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"mutex_wait_us":22,"thread_start_us":81,"threads_started":1}
I20260925 04:48:24.450302 20598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:24.444201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46524 (local address 127.0.98.66:34437)
0925 04:48:24.444593 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:24.444606 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:24.444627 (+ 21us) server_negotiation.cc:407] Connection header received
0925 04:48:24.444738 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:24.444742 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:24.444892 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:24.445054 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:48:24.445852 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:24.446965 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:24.448361 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:24.449127 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:24.449237 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:24.449412 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:24.449731 (+ 319us) server_negotiation.cc:299] Negotiation successful
0925 04:48:24.449828 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":107,"threads_started":1}
I20260925 04:48:24.450451 20593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:24.443301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:39455 (local address 127.0.0.1:44954)
0925 04:48:24.443986 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:24.444003 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 04:48:24.444082 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:48:24.445065 (+ 983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:48:24.445071 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:48:24.445106 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:48:24.445496 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:24.445502 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:24.446879 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:24.446883 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:24.448139 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:24.448150 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:24.449037 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:24.449040 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:24.449132 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:24.449153 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 04:48:24.449206 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 04:48:24.449258 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"mutex_wait_us":28,"spinlock_wait_cycles":5248,"thread_start_us":130,"threads_started":1}
I20260925 04:48:24.453048 20385 tablet_service.cc:1467] Processing CreateTablet for tablet 3dfecab2dae84f16ba32c2cb9b89db4c (DEFAULT_TABLE table=test-table [id=16704d61d6b744fba054c452404d3b0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:48:24.453148 20516 tablet_service.cc:1467] Processing CreateTablet for tablet 3dfecab2dae84f16ba32c2cb9b89db4c (DEFAULT_TABLE table=test-table [id=16704d61d6b744fba054c452404d3b0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:48:24.453145 20254 tablet_service.cc:1467] Processing CreateTablet for tablet 3dfecab2dae84f16ba32c2cb9b89db4c (DEFAULT_TABLE table=test-table [id=16704d61d6b744fba054c452404d3b0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:48:24.454150 20385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3dfecab2dae84f16ba32c2cb9b89db4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:48:24.454151 20516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3dfecab2dae84f16ba32c2cb9b89db4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:48:24.454151 20254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3dfecab2dae84f16ba32c2cb9b89db4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:48:24.460704 20599 tablet_bootstrap.cc:492] T 3dfecab2dae84f16ba32c2cb9b89db4c P d4e70b1e2be640f9ac5f4d567d70bfa9: Bootstrap starting.
I20260925 04:48:24.462146 20600 tablet_bootstrap.cc:492] T 3dfecab2dae84f16ba32c2cb9b89db4c P 50a302a85e9942bca33a9de7abe5ff3a: Bootstrap starting.
I20260925 04:48:24.462314 20601 tablet_bootstrap.cc:492] T 3dfecab2dae84f16ba32c2cb9b89db4c P 359a9cda7084405a9a9bc572f991145b: Bootstrap starting.
I20260925 04:48:24.463277 20599 tablet_bootstrap.cc:654] T 3dfecab2dae84f16ba32c2cb9b89db4c P d4e70b1e2be640f9ac5f4d567d70bfa9: Neither blocks nor log segments found. Creating new log.
I20260925 04:48:24.464020 20599 log.cc:824] T 3dfecab2dae84f16ba32c2cb9b89db4c P d4e70b1e2be640f9ac5f4d567d70bfa9: Log is configured to *not* fsync() on all Append() calls
I20260925 04:48:24.464466 20601 tablet_bootstrap.cc:654] T 3dfecab2dae84f16ba32c2cb9b89db4c P 359a9cda7084405a9a9bc572f991145b: Neither blocks nor log segments found. Creating new log.
I20260925 04:48:24.464466 20600 tablet_bootstrap.cc:654] T 3dfecab2dae84f16ba32c2cb9b89db4c P 50a302a85e9942bca33a9de7abe5ff3a: Neither blocks nor log segments found. Creating new log.
I20260925 04:48:24.465308 20600 log.cc:824] T 3dfecab2dae84f16ba32c2cb9b89db4c P 50a302a85e9942bca33a9de7abe5ff3a: Log is configured to *not* fsync() on all Append() calls
I20260925 04:48:24.465308 20601 log.cc:824] T 3dfecab2dae84f16ba32c2cb9b89db4c P 359a9cda7084405a9a9bc572f991145b: Log is configured to *not* fsync() on all Append() calls
I20260925 04:48:24.466432 20599 tablet_bootstrap.cc:492] T 3dfecab2dae84f16ba32c2cb9b89db4c P d4e70b1e2be640f9ac5f4d567d70bfa9: No bootstrap required, opened a new log
I20260925 04:48:24.466635 20599 ts_tablet_manager.cc:1397] T 3dfecab2dae84f16ba32c2cb9b89db4c P d4e70b1e2be640f9ac5f4d567d70bfa9: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 04:48:24.467152 20600 tablet_bootstrap.cc:492] T 3dfecab2dae84f16ba32c2cb9b89db4c P 50a302a85e9942bca33a9de7abe5ff3a: No bootstrap required, opened a new log
I20260925 04:48:24.467355 20600 ts_tablet_manager.cc:1397] T 3dfecab2dae84f16ba32c2cb9b89db4c P 50a302a85e9942bca33a9de7abe5ff3a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 04:48:24.467389 20601 tablet_bootstrap.cc:492] T 3dfecab2dae84f16ba32c2cb9b89db4c P 359a9cda7084405a9a9bc572f991145b: No bootstrap required, opened a new log
I20260925 04:48:24.467556 20601 ts_tablet_manager.cc:1397] T 3dfecab2dae84f16ba32c2cb9b89db4c P 359a9cda7084405a9a9bc572f991145b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 04:48:24.470497 20600 raft_consensus.cc:348] T 3dfecab2dae84f16ba32c2cb9b89db4c P 50a302a85e9942bca33a9de7abe5ff3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50a302a85e9942bca33a9de7abe5ff3a" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41765 } } peers { permanent_uuid: "359a9cda7084405a9a9bc572f991145b" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 39455 } } peers { permanent_uuid: "d4e70b1e2be640f9ac5f4d567d70bfa9" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 34437 } }
I20260925 04:48:24.470749 20600 raft_consensus.cc:374] T 3dfecab2dae84f16ba32c2cb9b89db4c P 50a302a85e9942bca33a9de7abe5ff3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:48:24.470803 20600 raft_consensus.cc:729] T 3dfecab2dae84f16ba32c2cb9b89db4c P 50a302a85e9942bca33a9de7abe5ff3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50a302a85e9942bca33a9de7abe5ff3a, State: Initialized, Role: FOLLOWER
I20260925 04:48:24.471072 20599 raft_consensus.cc:348] T 3dfecab2dae84f16ba32c2cb9b89db4c P d4e70b1e2be640f9ac5f4d567d70bfa9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50a302a85e9942bca33a9de7abe5ff3a" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41765 } } peers { permanent_uuid: "359a9cda7084405a9a9bc572f991145b" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 39455 } } peers { permanent_uuid: "d4e70b1e2be640f9ac5f4d567d70bfa9" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 34437 } }
I20260925 04:48:24.471243 20600 consensus_queue.cc:260] T 3dfecab2dae84f16ba32c2cb9b89db4c P 50a302a85e9942bca33a9de7abe5ff3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50a302a85e9942bca33a9de7abe5ff3a" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41765 } } peers { permanent_uuid: "359a9cda7084405a9a9bc572f991145b" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 39455 } } peers { permanent_uuid: "d4e70b1e2be640f9ac5f4d567d70bfa9" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 34437 } }
I20260925 04:48:24.471338 20599 raft_consensus.cc:374] T 3dfecab2dae84f16ba32c2cb9b89db4c P d4e70b1e2be640f9ac5f4d567d70bfa9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:48:24.471400 20599 raft_consensus.cc:729] T 3dfecab2dae84f16ba32c2cb9b89db4c P d4e70b1e2be640f9ac5f4d567d70bfa9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4e70b1e2be640f9ac5f4d567d70bfa9, State: Initialized, Role: FOLLOWER
I20260925 04:48:24.471905 20320 heartbeater.cc:503] Master 127.0.98.126:37957 was elected leader, sending a full tablet report...
I20260925 04:48:24.471849 20599 consensus_queue.cc:260] T 3dfecab2dae84f16ba32c2cb9b89db4c P d4e70b1e2be640f9ac5f4d567d70bfa9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50a302a85e9942bca33a9de7abe5ff3a" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41765 } } peers { permanent_uuid: "359a9cda7084405a9a9bc572f991145b" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 39455 } } peers { permanent_uuid: "d4e70b1e2be640f9ac5f4d567d70bfa9" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 34437 } }
I20260925 04:48:24.472333 20600 ts_tablet_manager.cc:1428] T 3dfecab2dae84f16ba32c2cb9b89db4c P 50a302a85e9942bca33a9de7abe5ff3a: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260925 04:48:24.472841 20599 ts_tablet_manager.cc:1428] T 3dfecab2dae84f16ba32c2cb9b89db4c P d4e70b1e2be640f9ac5f4d567d70bfa9: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20260925 04:48:24.472777 20601 raft_consensus.cc:348] T 3dfecab2dae84f16ba32c2cb9b89db4c P 359a9cda7084405a9a9bc572f991145b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50a302a85e9942bca33a9de7abe5ff3a" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41765 } } peers { permanent_uuid: "359a9cda7084405a9a9bc572f991145b" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 39455 } } peers { permanent_uuid: "d4e70b1e2be640f9ac5f4d567d70bfa9" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 34437 } }
I20260925 04:48:24.473027 20601 raft_consensus.cc:374] T 3dfecab2dae84f16ba32c2cb9b89db4c P 359a9cda7084405a9a9bc572f991145b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:48:24.473083 20601 raft_consensus.cc:729] T 3dfecab2dae84f16ba32c2cb9b89db4c P 359a9cda7084405a9a9bc572f991145b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 359a9cda7084405a9a9bc572f991145b, State: Initialized, Role: FOLLOWER
I20260925 04:48:24.473173 20451 heartbeater.cc:503] Master 127.0.98.126:37957 was elected leader, sending a full tablet report...
I20260925 04:48:24.473476 20601 consensus_queue.cc:260] T 3dfecab2dae84f16ba32c2cb9b89db4c P 359a9cda7084405a9a9bc572f991145b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50a302a85e9942bca33a9de7abe5ff3a" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41765 } } peers { permanent_uuid: "359a9cda7084405a9a9bc572f991145b" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 39455 } } peers { permanent_uuid: "d4e70b1e2be640f9ac5f4d567d70bfa9" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 34437 } }
I20260925 04:48:24.474035 20582 heartbeater.cc:503] Master 127.0.98.126:37957 was elected leader, sending a full tablet report...
I20260925 04:48:24.474426 20601 ts_tablet_manager.cc:1428] T 3dfecab2dae84f16ba32c2cb9b89db4c P 359a9cda7084405a9a9bc572f991145b: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
W20260925 04:48:24.514376 20321 tablet.cc:2369] T 3dfecab2dae84f16ba32c2cb9b89db4c P 50a302a85e9942bca33a9de7abe5ff3a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:48:24.587708 20606 raft_consensus.cc:482] T 3dfecab2dae84f16ba32c2cb9b89db4c P d4e70b1e2be640f9ac5f4d567d70bfa9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:48:24.588022 20606 raft_consensus.cc:504] T 3dfecab2dae84f16ba32c2cb9b89db4c P d4e70b1e2be640f9ac5f4d567d70bfa9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50a302a85e9942bca33a9de7abe5ff3a" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41765 } } peers { permanent_uuid: "359a9cda7084405a9a9bc572f991145b" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 39455 } } peers { permanent_uuid: "d4e70b1e2be640f9ac5f4d567d70bfa9" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 34437 } }
I20260925 04:48:24.589380 20606 leader_election.cc:288] T 3dfecab2dae84f16ba32c2cb9b89db4c P d4e70b1e2be640f9ac5f4d567d70bfa9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 50a302a85e9942bca33a9de7abe5ff3a (127.0.98.65:41765), 359a9cda7084405a9a9bc572f991145b (127.0.98.67:39455)
I20260925 04:48:24.593900 20595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:24.589752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:60911 (local address 127.0.98.65:41765)
0925 04:48:24.589882 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:24.589887 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:24.589904 (+ 17us) server_negotiation.cc:407] Connection header received
0925 04:48:24.590098 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:24.590103 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:24.590174 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:24.590295 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:48:24.590704 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:24.591280 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:24.592622 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:24.593236 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:24.593352 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:24.593419 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:24.593688 (+ 269us) server_negotiation.cc:299] Negotiation successful
0925 04:48:24.593763 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 04:48:24.593911 20597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:24.589805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:42839 (local address 127.0.98.67:39455)
0925 04:48:24.590038 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:24.590045 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:24.590188 (+ 143us) server_negotiation.cc:407] Connection header received
0925 04:48:24.590434 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:24.590437 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:24.590483 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:24.590570 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:48:24.590880 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:24.591400 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:24.592633 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:24.593249 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:24.593352 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:24.593419 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:24.593688 (+ 269us) server_negotiation.cc:299] Negotiation successful
0925 04:48:24.593763 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260925 04:48:24.593923 20453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:24.589608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:41765 (local address 127.0.98.66:60911)
0925 04:48:24.589822 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:24.589839 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 04:48:24.589946 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:48:24.590413 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:48:24.590416 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:48:24.590435 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:48:24.590576 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:24.590584 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:24.591394 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:24.591398 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:24.592471 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:24.592480 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:24.593365 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:24.593369 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:24.593536 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:24.593580 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 04:48:24.593657 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 04:48:24.593759 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":149}
I20260925 04:48:24.593931 20608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:24.589683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:39455 (local address 127.0.98.66:42839)
0925 04:48:24.590084 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:24.590099 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:48:24.590207 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:48:24.590590 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:48:24.590593 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:48:24.590608 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:48:24.590764 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:24.590771 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:24.591522 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:24.591525 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:24.592458 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:24.592473 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:24.593365 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:24.593369 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:24.593533 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:24.593577 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 04:48:24.593657 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 04:48:24.593751 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"mutex_wait_us":62,"thread_start_us":142,"threads_started":1}
I20260925 04:48:24.594683 20274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dfecab2dae84f16ba32c2cb9b89db4c" candidate_uuid: "d4e70b1e2be640f9ac5f4d567d70bfa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50a302a85e9942bca33a9de7abe5ff3a" is_pre_election: true
I20260925 04:48:24.594944 20536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dfecab2dae84f16ba32c2cb9b89db4c" candidate_uuid: "d4e70b1e2be640f9ac5f4d567d70bfa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "359a9cda7084405a9a9bc572f991145b" is_pre_election: true
I20260925 04:48:24.595031 20274 raft_consensus.cc:2454] T 3dfecab2dae84f16ba32c2cb9b89db4c P 50a302a85e9942bca33a9de7abe5ff3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4e70b1e2be640f9ac5f4d567d70bfa9 in term 0.
I20260925 04:48:24.595242 20536 raft_consensus.cc:2454] T 3dfecab2dae84f16ba32c2cb9b89db4c P 359a9cda7084405a9a9bc572f991145b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4e70b1e2be640f9ac5f4d567d70bfa9 in term 0.
I20260925 04:48:24.595588 20339 leader_election.cc:302] T 3dfecab2dae84f16ba32c2cb9b89db4c P d4e70b1e2be640f9ac5f4d567d70bfa9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50a302a85e9942bca33a9de7abe5ff3a, d4e70b1e2be640f9ac5f4d567d70bfa9; no voters:
I20260925 04:48:24.595916 20606 raft_consensus.cc:2781] T 3dfecab2dae84f16ba32c2cb9b89db4c P d4e70b1e2be640f9ac5f4d567d70bfa9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:48:24.596006 20606 raft_consensus.cc:482] T 3dfecab2dae84f16ba32c2cb9b89db4c P d4e70b1e2be640f9ac5f4d567d70bfa9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:48:24.596050 20606 raft_consensus.cc:3037] T 3dfecab2dae84f16ba32c2cb9b89db4c P d4e70b1e2be640f9ac5f4d567d70bfa9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:48:24.597057 20606 raft_consensus.cc:504] T 3dfecab2dae84f16ba32c2cb9b89db4c P d4e70b1e2be640f9ac5f4d567d70bfa9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50a302a85e9942bca33a9de7abe5ff3a" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41765 } } peers { permanent_uuid: "359a9cda7084405a9a9bc572f991145b" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 39455 } } peers { permanent_uuid: "d4e70b1e2be640f9ac5f4d567d70bfa9" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 34437 } }
I20260925 04:48:24.597424 20606 leader_election.cc:288] T 3dfecab2dae84f16ba32c2cb9b89db4c P d4e70b1e2be640f9ac5f4d567d70bfa9 [CANDIDATE]: Term 1 election: Requested vote from peers 50a302a85e9942bca33a9de7abe5ff3a (127.0.98.65:41765), 359a9cda7084405a9a9bc572f991145b (127.0.98.67:39455)
I20260925 04:48:24.597776 20274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dfecab2dae84f16ba32c2cb9b89db4c" candidate_uuid: "d4e70b1e2be640f9ac5f4d567d70bfa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50a302a85e9942bca33a9de7abe5ff3a"
I20260925 04:48:24.597807 20536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dfecab2dae84f16ba32c2cb9b89db4c" candidate_uuid: "d4e70b1e2be640f9ac5f4d567d70bfa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "359a9cda7084405a9a9bc572f991145b"
I20260925 04:48:24.597899 20274 raft_consensus.cc:3037] T 3dfecab2dae84f16ba32c2cb9b89db4c P 50a302a85e9942bca33a9de7abe5ff3a [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:48:24.597965 20536 raft_consensus.cc:3037] T 3dfecab2dae84f16ba32c2cb9b89db4c P 359a9cda7084405a9a9bc572f991145b [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:48:24.599054 20274 raft_consensus.cc:2454] T 3dfecab2dae84f16ba32c2cb9b89db4c P 50a302a85e9942bca33a9de7abe5ff3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4e70b1e2be640f9ac5f4d567d70bfa9 in term 1.
I20260925 04:48:24.599077 20536 raft_consensus.cc:2454] T 3dfecab2dae84f16ba32c2cb9b89db4c P 359a9cda7084405a9a9bc572f991145b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4e70b1e2be640f9ac5f4d567d70bfa9 in term 1.
I20260925 04:48:24.599359 20339 leader_election.cc:302] T 3dfecab2dae84f16ba32c2cb9b89db4c P d4e70b1e2be640f9ac5f4d567d70bfa9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50a302a85e9942bca33a9de7abe5ff3a, d4e70b1e2be640f9ac5f4d567d70bfa9; no voters:
I20260925 04:48:24.599602 20606 raft_consensus.cc:2781] T 3dfecab2dae84f16ba32c2cb9b89db4c P d4e70b1e2be640f9ac5f4d567d70bfa9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:48:24.599855 20606 raft_consensus.cc:686] T 3dfecab2dae84f16ba32c2cb9b89db4c P d4e70b1e2be640f9ac5f4d567d70bfa9 [term 1 LEADER]: Becoming Leader. State: Replica: d4e70b1e2be640f9ac5f4d567d70bfa9, State: Running, Role: LEADER
I20260925 04:48:24.600226 20606 consensus_queue.cc:237] T 3dfecab2dae84f16ba32c2cb9b89db4c P d4e70b1e2be640f9ac5f4d567d70bfa9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50a302a85e9942bca33a9de7abe5ff3a" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41765 } } peers { permanent_uuid: "359a9cda7084405a9a9bc572f991145b" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 39455 } } peers { permanent_uuid: "d4e70b1e2be640f9ac5f4d567d70bfa9" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 34437 } }
I20260925 04:48:24.603385 20132 catalog_manager.cc:5495] T 3dfecab2dae84f16ba32c2cb9b89db4c P d4e70b1e2be640f9ac5f4d567d70bfa9 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4e70b1e2be640f9ac5f4d567d70bfa9 (127.0.98.66). New cstate: current_term: 1 leader_uuid: "d4e70b1e2be640f9ac5f4d567d70bfa9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50a302a85e9942bca33a9de7abe5ff3a" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "359a9cda7084405a9a9bc572f991145b" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 39455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4e70b1e2be640f9ac5f4d567d70bfa9" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 34437 } health_report { overall_health: HEALTHY } } }
I20260925 04:48:24.638931 20598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:24.635235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46536 (local address 127.0.98.66:34437)
0925 04:48:24.635399 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:24.635407 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:24.635424 (+ 17us) server_negotiation.cc:407] Connection header received
0925 04:48:24.635512 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:24.635514 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:24.635568 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:24.635688 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:48:24.635936 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:24.636457 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:24.637351 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:24.637582 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:24.637707 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:24.638026 (+ 319us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:48:24.638509 (+ 483us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:48:24.638654 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:24.638696 (+ 42us) server_negotiation.cc:299] Negotiation successful
0925 04:48:24.638751 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 04:48:24.645030 20274 raft_consensus.cc:1261] T 3dfecab2dae84f16ba32c2cb9b89db4c P 50a302a85e9942bca33a9de7abe5ff3a [term 1 FOLLOWER]: Refusing update from remote peer d4e70b1e2be640f9ac5f4d567d70bfa9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:48:24.645030 20536 raft_consensus.cc:1261] T 3dfecab2dae84f16ba32c2cb9b89db4c P 359a9cda7084405a9a9bc572f991145b [term 1 FOLLOWER]: Refusing update from remote peer d4e70b1e2be640f9ac5f4d567d70bfa9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:48:24.645839 20606 consensus_queue.cc:1035] T 3dfecab2dae84f16ba32c2cb9b89db4c P d4e70b1e2be640f9ac5f4d567d70bfa9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50a302a85e9942bca33a9de7abe5ff3a" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 41765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:48:24.646051 20609 consensus_queue.cc:1035] T 3dfecab2dae84f16ba32c2cb9b89db4c P d4e70b1e2be640f9ac5f4d567d70bfa9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "359a9cda7084405a9a9bc572f991145b" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 39455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:48:24.654853 20614 mvcc.cc:205] Tried to move back new op lower bound from 7333116742218530816 to 7333116742045208576. Current Snapshot: MvccSnapshot[applied={T|T < 7333116742218530816}]
I20260925 04:48:24.663611 20131 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47864:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:48:24.663841 20131 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47864:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:48:24.666445 20131 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4b81bff3f581432b9157c60a631dcc92: Sending DeleteTablet for 3 replicas of tablet 3dfecab2dae84f16ba32c2cb9b89db4c
I20260925 04:48:24.667356 20254 tablet_service.cc:1514] Processing DeleteTablet for tablet 3dfecab2dae84f16ba32c2cb9b89db4c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:48:24 UTC) from {username='slave'} at 127.0.0.1:59662
I20260925 04:48:24.667380 20516 tablet_service.cc:1514] Processing DeleteTablet for tablet 3dfecab2dae84f16ba32c2cb9b89db4c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:48:24 UTC) from {username='slave'} at 127.0.0.1:44954
I20260925 04:48:24.667841 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 20192
I20260925 04:48:24.668555 20385 tablet_service.cc:1514] Processing DeleteTablet for tablet 3dfecab2dae84f16ba32c2cb9b89db4c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:48:24 UTC) from {username='slave'} at 127.0.0.1:46524
I20260925 04:48:24.668974 20624 tablet_replica.cc:330] T 3dfecab2dae84f16ba32c2cb9b89db4c P d4e70b1e2be640f9ac5f4d567d70bfa9: stopping tablet replica
I20260925 04:48:24.669277 20624 raft_consensus.cc:2229] T 3dfecab2dae84f16ba32c2cb9b89db4c P d4e70b1e2be640f9ac5f4d567d70bfa9 [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:48:24.669637 20624 raft_consensus.cc:2258] T 3dfecab2dae84f16ba32c2cb9b89db4c P d4e70b1e2be640f9ac5f4d567d70bfa9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:48:24.671188 20624 ts_tablet_manager.cc:1905] T 3dfecab2dae84f16ba32c2cb9b89db4c P d4e70b1e2be640f9ac5f4d567d70bfa9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:48:24.674015 20624 ts_tablet_manager.cc:1918] T 3dfecab2dae84f16ba32c2cb9b89db4c P d4e70b1e2be640f9ac5f4d567d70bfa9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:48:24.674121 20624 log.cc:1196] T 3dfecab2dae84f16ba32c2cb9b89db4c P d4e70b1e2be640f9ac5f4d567d70bfa9: Deleting WAL directory at /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311449873301-393-0/minicluster-data/ts-1/wal/wals/3dfecab2dae84f16ba32c2cb9b89db4c
I20260925 04:48:24.674482 20624 ts_tablet_manager.cc:1939] T 3dfecab2dae84f16ba32c2cb9b89db4c P d4e70b1e2be640f9ac5f4d567d70bfa9: Deleting consensus metadata
W20260925 04:48:24.675773 20117 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.65:41765 (error 108)
I20260925 04:48:24.675907 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 20323
I20260925 04:48:24.676017 20623 tablet_replica.cc:330] T 3dfecab2dae84f16ba32c2cb9b89db4c P 359a9cda7084405a9a9bc572f991145b: stopping tablet replica
I20260925 04:48:24.676373 20623 raft_consensus.cc:2229] T 3dfecab2dae84f16ba32c2cb9b89db4c P 359a9cda7084405a9a9bc572f991145b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:48:24.676743 20623 raft_consensus.cc:2258] T 3dfecab2dae84f16ba32c2cb9b89db4c P 359a9cda7084405a9a9bc572f991145b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:48:24.678296 20623 ts_tablet_manager.cc:1905] T 3dfecab2dae84f16ba32c2cb9b89db4c P 359a9cda7084405a9a9bc572f991145b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:48:24.680657 20623 ts_tablet_manager.cc:1918] T 3dfecab2dae84f16ba32c2cb9b89db4c P 359a9cda7084405a9a9bc572f991145b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:48:24.680751 20623 log.cc:1196] T 3dfecab2dae84f16ba32c2cb9b89db4c P 359a9cda7084405a9a9bc572f991145b: Deleting WAL directory at /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790311449873301-393-0/minicluster-data/ts-2/wal/wals/3dfecab2dae84f16ba32c2cb9b89db4c
I20260925 04:48:24.681069 20623 ts_tablet_manager.cc:1939] T 3dfecab2dae84f16ba32c2cb9b89db4c P 359a9cda7084405a9a9bc572f991145b: Deleting consensus metadata
I20260925 04:48:24.681860 20593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:24.681520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:41765 (local address 127.0.0.1:59676)
0925 04:48:24.681680 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:24.681766 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.65:41765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260925 04:48:24.682119 20117 catalog_manager.cc:4584] TS 50a302a85e9942bca33a9de7abe5ff3a (127.0.98.65:41765): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3dfecab2dae84f16ba32c2cb9b89db4c: Network error: Client connection negotiation failed: client connection to 127.0.98.65:41765: connect: Connection refused (error 111)
I20260925 04:48:24.682407 20116 catalog_manager.cc:4841] TS 359a9cda7084405a9a9bc572f991145b (127.0.98.67:39455): tablet 3dfecab2dae84f16ba32c2cb9b89db4c (table test-table [id=16704d61d6b744fba054c452404d3b0f]) successfully deleted
I20260925 04:48:24.683708 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 20454
I20260925 04:48:24.684577 20593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:24.683897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:34437 (local address 127.0.0.1:46538)
0925 04:48:24.684470 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:24.684513 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.66:34437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":510}
I20260925 04:48:24.690554 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 20101
2026-09-25T04:48:24Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1161 ms)
[----------] 6 tests from EncryptionPolicyTest (10304 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790311449873301-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:48:24 dist-test-slave-kghz krb5kdc[20628](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:48:24 dist-test-slave-kghz krb5kdc[20628](info): set up 2 sockets
Sep 25 04:48:24 dist-test-slave-kghz krb5kdc[20628](info): commencing operation
krb5kdc: starting...
W20260925 04:48:25.722543 393 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790311449873301-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790311449873301-393-0/krb5kdc/test-user.keytab.
Sep 25 04:48:25 dist-test-slave-kghz krb5kdc[20628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311705, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:48:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:48:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790311449873301-393-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790311449873301-393-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 04:48:25.849699 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790311449873301-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790311449873301-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790311449873301-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40959
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790311449873301-393-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40959
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:48:25.964363 20641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:48:25.964676 20641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:48:25.964772 20641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:48:25.968858 20641 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:48:25.968958 20641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:48:25.968997 20641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:48:25.969034 20641 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:48:25.969070 20641 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:48:25.974329 20641 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790311449873301-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790311449873301-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40959
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790311449873301-393-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20641
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790311449873301-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:48:25.975477 20641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:48:25.976536 20641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:48:25.985471 20647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:48:25.985414 20646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:48:25.985522 20641 server_base.cc:1034] running on GCE node
W20260925 04:48:25.985438 20649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:48:25.986338 20641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:48:25.987510 20641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:48:25.988708 20641 hybrid_clock.cc:648] HybridClock initialized: now 1790311705988668 us; error 60 us; skew 500 ppm
Sep 25 04:48:25 dist-test-slave-kghz krb5kdc[20628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311705, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:48:25.991662 20641 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 04:48:25.992614 20641 webserver.cc:460] Webserver started at http://127.0.0.1:44053/ using document root <none> and password file <none>
I20260925 04:48:25.993199 20641 fs_manager.cc:359] Metadata directory not provided
I20260925 04:48:25.993292 20641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:48:25.993546 20641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:48:25.995733 20641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790311449873301-393-0/minicluster-data/master-0/data/instance:
uuid: "9dc6d2574f584a029c27494592dc6a24"
format_stamp: "Formatted at 2026-09-25 04:48:25 on dist-test-slave-kghz"
server_key: "60920813199510b9d672bafd4aa234ff"
server_key_iv: "0f96234045abb1df506195ed203978cd"
server_key_version: "encryptionkey@0"
I20260925 04:48:25.996313 20641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790311449873301-393-0/minicluster-data/master-0/wal/instance:
uuid: "9dc6d2574f584a029c27494592dc6a24"
format_stamp: "Formatted at 2026-09-25 04:48:25 on dist-test-slave-kghz"
server_key: "60920813199510b9d672bafd4aa234ff"
server_key_iv: "0f96234045abb1df506195ed203978cd"
server_key_version: "encryptionkey@0"
I20260925 04:48:26.000170 20641 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 04:48:26.002640 20656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:48:26.003999 20641 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 04:48:26.004160 20641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790311449873301-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790311449873301-393-0/minicluster-data/master-0/wal
uuid: "9dc6d2574f584a029c27494592dc6a24"
format_stamp: "Formatted at 2026-09-25 04:48:25 on dist-test-slave-kghz"
server_key: "60920813199510b9d672bafd4aa234ff"
server_key_iv: "0f96234045abb1df506195ed203978cd"
server_key_version: "encryptionkey@0"
I20260925 04:48:26.004284 20641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790311449873301-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790311449873301-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790311449873301-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:48:26.016443 20641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:48:26.019582 20641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:48:26.019795 20641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:48:26.027083 20641 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40959
I20260925 04:48:26.027105 20708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40959 every 8 connection(s)
I20260925 04:48:26.028087 20641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
I20260925 04:48:26.030957 20709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:48:26.036401 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 20641
I20260925 04:48:26.036514 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790311449873301-393-0/minicluster-data/master-0/wal/instance
I20260925 04:48:26.036654 20709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dc6d2574f584a029c27494592dc6a24: Bootstrap starting.
I20260925 04:48:26.036839 393 external_mini_cluster.cc:1428] Setting key 4ab8223933bf3a93fc5890d760881ed5
I20260925 04:48:26.039270 20709 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9dc6d2574f584a029c27494592dc6a24: Neither blocks nor log segments found. Creating new log.
I20260925 04:48:26.040045 20709 log.cc:824] T 00000000000000000000000000000000 P 9dc6d2574f584a029c27494592dc6a24: Log is configured to *not* fsync() on all Append() calls
I20260925 04:48:26.041906 20709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dc6d2574f584a029c27494592dc6a24: No bootstrap required, opened a new log
Sep 25 04:48:26 dist-test-slave-kghz krb5kdc[20628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311705, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 04:48:26.045773 20709 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9dc6d2574f584a029c27494592dc6a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dc6d2574f584a029c27494592dc6a24" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40959 } }
I20260925 04:48:26.046079 20709 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9dc6d2574f584a029c27494592dc6a24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:48:26.046141 20709 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9dc6d2574f584a029c27494592dc6a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dc6d2574f584a029c27494592dc6a24, State: Initialized, Role: FOLLOWER
I20260925 04:48:26.046684 20709 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9dc6d2574f584a029c27494592dc6a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dc6d2574f584a029c27494592dc6a24" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40959 } }
I20260925 04:48:26.046824 20709 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9dc6d2574f584a029c27494592dc6a24 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:48:26.046923 20709 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9dc6d2574f584a029c27494592dc6a24 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:48:26.047061 20709 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9dc6d2574f584a029c27494592dc6a24 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:48:26.048208 20709 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9dc6d2574f584a029c27494592dc6a24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dc6d2574f584a029c27494592dc6a24" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40959 } }
I20260925 04:48:26.048512 20709 leader_election.cc:302] T 00000000000000000000000000000000 P 9dc6d2574f584a029c27494592dc6a24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9dc6d2574f584a029c27494592dc6a24; no voters:
I20260925 04:48:26.048774 20709 leader_election.cc:288] T 00000000000000000000000000000000 P 9dc6d2574f584a029c27494592dc6a24 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:48:26.048900 20714 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9dc6d2574f584a029c27494592dc6a24 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:48:26.049181 20714 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9dc6d2574f584a029c27494592dc6a24 [term 1 LEADER]: Becoming Leader. State: Replica: 9dc6d2574f584a029c27494592dc6a24, State: Running, Role: LEADER
I20260925 04:48:26.049624 20714 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9dc6d2574f584a029c27494592dc6a24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dc6d2574f584a029c27494592dc6a24" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40959 } }
I20260925 04:48:26.050284 20709 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9dc6d2574f584a029c27494592dc6a24 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:48:26.052248 20712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:26.038336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59717 (local address 127.0.0.1:40959)
0925 04:48:26.038778 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:26.038793 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:26.038828 (+ 35us) server_negotiation.cc:407] Connection header received
0925 04:48:26.039539 (+ 711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:26.039560 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:26.039909 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:26.040257 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:26.041057 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:26.042075 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:26.042996 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:26.043498 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:26.043670 (+ 172us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:26.046659 (+ 2989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:26.046664 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:26.046678 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:48:26.046708 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:26.048692 (+ 1984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:26.049353 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:26.049359 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:26.049376 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:26.049515 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:26.049892 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:26.049896 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:26.049898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:26.050466 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:26.050688 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:26.051024 (+ 336us) server_negotiation.cc:299] Negotiation successful
0925 04:48:26.051256 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":136,"threads_started":1}
I20260925 04:48:26.053215 20715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dc6d2574f584a029c27494592dc6a24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9dc6d2574f584a029c27494592dc6a24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dc6d2574f584a029c27494592dc6a24" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40959 } } }
I20260925 04:48:26.053267 20716 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dc6d2574f584a029c27494592dc6a24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9dc6d2574f584a029c27494592dc6a24. Latest consensus state: current_term: 1 leader_uuid: "9dc6d2574f584a029c27494592dc6a24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dc6d2574f584a029c27494592dc6a24" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40959 } } }
I20260925 04:48:26.053334 20715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dc6d2574f584a029c27494592dc6a24 [sys.catalog]: This master's current role is: LEADER
I20260925 04:48:26.053360 20716 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dc6d2574f584a029c27494592dc6a24 [sys.catalog]: This master's current role is: LEADER
I20260925 04:48:26.055971 393 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260925 04:48:26.055971 20729 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9dc6d2574f584a029c27494592dc6a24: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 04:48:26.056059 20729 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 04:48:26.056161 20730 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:48:26.056842 20730 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:48:26.063227 20730 catalog_manager.cc:1347] Generated new cluster ID: 5532d3500c024850b5d4aac42cdf75d6
I20260925 04:48:26.063293 20730 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:48:26.068377 20712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:26.058355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36839 (local address 127.0.0.1:40959)
0925 04:48:26.058476 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:26.058480 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:26.059123 (+ 643us) server_negotiation.cc:407] Connection header received
0925 04:48:26.059315 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:26.059319 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:26.059368 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:26.059421 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:26.060336 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:26.061567 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:26.062458 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:26.062689 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:26.062801 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:26.064682 (+ 1881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:26.064686 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:26.064689 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:48:26.064716 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:26.066453 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:26.066995 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:26.066998 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:26.067000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:26.067089 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:26.067435 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:26.067438 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:26.067440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:26.067660 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:26.067834 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:26.068089 (+ 255us) server_negotiation.cc:299] Negotiation successful
0925 04:48:26.068186 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 04:48:26.080816 20730 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:48:26.081773 20730 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:48:26.086825 20730 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9dc6d2574f584a029c27494592dc6a24: Generated new TSK 0
I20260925 04:48:26.087498 20730 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:48:26.094378 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 20641
W20260925 04:48:26.100024 20626 connection.cc:535] client connection to 127.0.0.1:40959 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40959): Connection reset by peer (error 104)
2026-09-25T04:48:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1402 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790311449873301-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:48:26 dist-test-slave-kghz krb5kdc[20740](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:48:26 dist-test-slave-kghz krb5kdc[20740](info): set up 2 sockets
Sep 25 04:48:26 dist-test-slave-kghz krb5kdc[20740](info): commencing operation
krb5kdc: starting...
W20260925 04:48:27.126560 393 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790311449873301-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790311449873301-393-0/krb5kdc/test-user.keytab.
Sep 25 04:48:27 dist-test-slave-kghz krb5kdc[20740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311707, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:48:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:48:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790311449873301-393-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790311449873301-393-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 04:48:27.253531 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790311449873301-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790311449873301-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790311449873301-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37117
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790311449873301-393-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37117
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:48:27.363940 20753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:48:27.364202 20753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:48:27.364243 20753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:48:27.368350 20753 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:48:27.368415 20753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:48:27.368435 20753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:48:27.368453 20753 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:48:27.368468 20753 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:48:27.373836 20753 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790311449873301-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790311449873301-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37117
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790311449873301-393-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20753
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790311449873301-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:48:27.374967 20753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:48:27.375993 20753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:48:27.384626 20758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:48:27.384644 20759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:48:27.384634 20761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:48:27.384896 20753 server_base.cc:1034] running on GCE node
I20260925 04:48:27.385535 20753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:48:27.386656 20753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:48:27.387821 20753 hybrid_clock.cc:648] HybridClock initialized: now 1790311707387804 us; error 31 us; skew 500 ppm
Sep 25 04:48:27 dist-test-slave-kghz krb5kdc[20740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311707, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:48:27.390542 20753 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 04:48:27.391347 20753 webserver.cc:460] Webserver started at http://127.0.0.1:42647/ using document root <none> and password file <none>
I20260925 04:48:27.391860 20753 fs_manager.cc:359] Metadata directory not provided
I20260925 04:48:27.391917 20753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:48:27.392074 20753 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:48:27.394179 20753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790311449873301-393-0/minicluster-data/master-0/data/instance:
uuid: "05d1fb2b755447a5b31479ff7fe14340"
format_stamp: "Formatted at 2026-09-25 04:48:27 on dist-test-slave-kghz"
server_key: "37e729ead28de849df62d384d30fbaec"
server_key_iv: "ab65c17a04d51983e6df3e9b09ac0e3d"
server_key_version: "encryptionkey@0"
I20260925 04:48:27.394747 20753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790311449873301-393-0/minicluster-data/master-0/wal/instance:
uuid: "05d1fb2b755447a5b31479ff7fe14340"
format_stamp: "Formatted at 2026-09-25 04:48:27 on dist-test-slave-kghz"
server_key: "37e729ead28de849df62d384d30fbaec"
server_key_iv: "ab65c17a04d51983e6df3e9b09ac0e3d"
server_key_version: "encryptionkey@0"
I20260925 04:48:27.398207 20753 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260925 04:48:27.400583 20768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:48:27.401882 20753 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 04:48:27.401983 20753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790311449873301-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790311449873301-393-0/minicluster-data/master-0/wal
uuid: "05d1fb2b755447a5b31479ff7fe14340"
format_stamp: "Formatted at 2026-09-25 04:48:27 on dist-test-slave-kghz"
server_key: "37e729ead28de849df62d384d30fbaec"
server_key_iv: "ab65c17a04d51983e6df3e9b09ac0e3d"
server_key_version: "encryptionkey@0"
I20260925 04:48:27.402067 20753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790311449873301-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790311449873301-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790311449873301-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:48:27.416539 20753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:48:27.419821 20753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:48:27.419975 20753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:48:27.426856 20753 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37117
I20260925 04:48:27.426872 20820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37117 every 8 connection(s)
I20260925 04:48:27.427810 20753 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
I20260925 04:48:27.429636 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 20753
I20260925 04:48:27.429796 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790311449873301-393-0/minicluster-data/master-0/wal/instance
I20260925 04:48:27.430088 393 external_mini_cluster.cc:1428] Setting key 1dcd03c0f8a7c263f548f9aef92590c6
I20260925 04:48:27.430949 20821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:48:27.437250 20821 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05d1fb2b755447a5b31479ff7fe14340: Bootstrap starting.
I20260925 04:48:27.439975 20824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:27.431586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45313 (local address 127.0.0.1:37117)
0925 04:48:27.432072 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:27.432086 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:27.432122 (+ 36us) server_negotiation.cc:407] Connection header received
0925 04:48:27.432765 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:27.432781 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:27.433369 (+ 588us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:27.433855 (+ 486us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:27.434315 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:27.435334 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:27.436257 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:27.436760 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:27.436932 (+ 172us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:27.437986 (+ 1054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:27.437991 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:27.438004 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:48:27.438624 (+ 620us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:27.438683 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:48:27.438708 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:27.438818 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:27.439034 (+ 216us) server_negotiation.cc:299] Negotiation successful
0925 04:48:27.439197 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":171,"threads_started":1}
I20260925 04:48:27.439981 20821 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05d1fb2b755447a5b31479ff7fe14340: Neither blocks nor log segments found. Creating new log.
I20260925 04:48:27.440763 20821 log.cc:824] T 00000000000000000000000000000000 P 05d1fb2b755447a5b31479ff7fe14340: Log is configured to *not* fsync() on all Append() calls
I20260925 04:48:27.442902 20821 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05d1fb2b755447a5b31479ff7fe14340: No bootstrap required, opened a new log
I20260925 04:48:27.443022 393 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:48:27.447309 20821 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05d1fb2b755447a5b31479ff7fe14340 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05d1fb2b755447a5b31479ff7fe14340" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37117 } }
I20260925 04:48:27.447619 20821 raft_consensus.cc:374] T 00000000000000000000000000000000 P 05d1fb2b755447a5b31479ff7fe14340 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:48:27.447671 20821 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05d1fb2b755447a5b31479ff7fe14340 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05d1fb2b755447a5b31479ff7fe14340, State: Initialized, Role: FOLLOWER
I20260925 04:48:27.448139 20821 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05d1fb2b755447a5b31479ff7fe14340 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05d1fb2b755447a5b31479ff7fe14340" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37117 } }
I20260925 04:48:27.448309 20821 raft_consensus.cc:388] T 00000000000000000000000000000000 P 05d1fb2b755447a5b31479ff7fe14340 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:48:27.448431 20821 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05d1fb2b755447a5b31479ff7fe14340 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:48:27.448552 20821 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05d1fb2b755447a5b31479ff7fe14340 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:48:27.449674 20824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:27.445142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34767 (local address 127.0.0.1:37117)
0925 04:48:27.445265 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:27.445268 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:27.445523 (+ 255us) server_negotiation.cc:407] Connection header received
0925 04:48:27.445669 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:27.445672 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:27.445714 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:27.445797 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:27.446138 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:27.446753 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:27.447891 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:27.448170 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:27.448259 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:27.449193 (+ 934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:27.449196 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:27.449198 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:48:27.449209 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:27.449233 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:48:27.449243 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:27.449311 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:27.449478 (+ 167us) server_negotiation.cc:299] Negotiation successful
0925 04:48:27.449531 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 04:48:27.449932 20821 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05d1fb2b755447a5b31479ff7fe14340 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05d1fb2b755447a5b31479ff7fe14340" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37117 } }
I20260925 04:48:27.450350 20821 leader_election.cc:302] T 00000000000000000000000000000000 P 05d1fb2b755447a5b31479ff7fe14340 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05d1fb2b755447a5b31479ff7fe14340; no voters:
I20260925 04:48:27.450726 20821 leader_election.cc:288] T 00000000000000000000000000000000 P 05d1fb2b755447a5b31479ff7fe14340 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:48:27.450817 20831 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05d1fb2b755447a5b31479ff7fe14340 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:48:27.451097 20831 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05d1fb2b755447a5b31479ff7fe14340 [term 1 LEADER]: Becoming Leader. State: Replica: 05d1fb2b755447a5b31479ff7fe14340, State: Running, Role: LEADER
I20260925 04:48:27.451433 20831 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05d1fb2b755447a5b31479ff7fe14340 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05d1fb2b755447a5b31479ff7fe14340" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37117 } }
I20260925 04:48:27.452034 20821 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05d1fb2b755447a5b31479ff7fe14340 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:48:27.453049 20833 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05d1fb2b755447a5b31479ff7fe14340 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05d1fb2b755447a5b31479ff7fe14340. Latest consensus state: current_term: 1 leader_uuid: "05d1fb2b755447a5b31479ff7fe14340" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05d1fb2b755447a5b31479ff7fe14340" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37117 } } }
I20260925 04:48:27.453169 20833 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05d1fb2b755447a5b31479ff7fe14340 [sys.catalog]: This master's current role is: LEADER
I20260925 04:48:27.453550 20840 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:48:27.453550 20832 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05d1fb2b755447a5b31479ff7fe14340 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "05d1fb2b755447a5b31479ff7fe14340" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05d1fb2b755447a5b31479ff7fe14340" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37117 } } }
I20260925 04:48:27.453729 20832 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05d1fb2b755447a5b31479ff7fe14340 [sys.catalog]: This master's current role is: LEADER
I20260925 04:48:27.457042 20840 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:48:27.463219 20840 catalog_manager.cc:1347] Generated new cluster ID: 166124920b384f639f23cefa017a84e5
I20260925 04:48:27.463291 20840 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:48:27.484822 20840 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:48:27.486155 20840 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:48:27.496141 20840 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 05d1fb2b755447a5b31479ff7fe14340: Generated new TSK 0
I20260925 04:48:27.496846 20840 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:48:27.509315 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 20753
W20260925 04:48:27.514956 20738 connection.cc:535] client connection to 127.0.0.1:37117 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37117): Connection reset by peer (error 104)
2026-09-25T04:48:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1414 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790311449873301-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:48:27 dist-test-slave-kghz krb5kdc[20852](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:48:27 dist-test-slave-kghz krb5kdc[20852](info): set up 2 sockets
Sep 25 04:48:27 dist-test-slave-kghz krb5kdc[20852](info): commencing operation
krb5kdc: starting...
W20260925 04:48:28.562615 393 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790311449873301-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790311449873301-393-0/krb5kdc/test-user.keytab.
Sep 25 04:48:28 dist-test-slave-kghz krb5kdc[20852](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311708, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:48:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:48:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790311449873301-393-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790311449873301-393-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 04:48:28.690212 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790311449873301-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790311449873301-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790311449873301-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34691
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790311449873301-393-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34691
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:48:28.800707 20865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:48:28.800969 20865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:48:28.801013 20865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:48:28.804996 20865 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:48:28.805061 20865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:48:28.805083 20865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:48:28.805099 20865 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:48:28.805115 20865 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:48:28.810591 20865 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790311449873301-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790311449873301-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34691
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790311449873301-393-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20865
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790311449873301-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:48:28.811654 20865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:48:28.812647 20865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:48:28.821698 20873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:48:28.821698 20870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:48:28.821734 20865 server_base.cc:1034] running on GCE node
W20260925 04:48:28.821714 20871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:48:28.822623 20865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:48:28.823729 20865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:48:28.824916 20865 hybrid_clock.cc:648] HybridClock initialized: now 1790311708824884 us; error 28 us; skew 500 ppm
Sep 25 04:48:28 dist-test-slave-kghz krb5kdc[20852](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311708, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:48:28.827787 20865 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 04:48:28.828701 20865 webserver.cc:460] Webserver started at http://127.0.0.1:33053/ using document root <none> and password file <none>
I20260925 04:48:28.829282 20865 fs_manager.cc:359] Metadata directory not provided
I20260925 04:48:28.829345 20865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:48:28.829558 20865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:48:28.831682 20865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790311449873301-393-0/minicluster-data/master-0/data/instance:
uuid: "b4907016da864d88911e1749421566b4"
format_stamp: "Formatted at 2026-09-25 04:48:28 on dist-test-slave-kghz"
server_key: "a67348942f8b1cc0405be056d5b616a0"
server_key_iv: "1b7ca834461565d92f3c857721ef0a7c"
server_key_version: "encryptionkey@0"
I20260925 04:48:28.832314 20865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790311449873301-393-0/minicluster-data/master-0/wal/instance:
uuid: "b4907016da864d88911e1749421566b4"
format_stamp: "Formatted at 2026-09-25 04:48:28 on dist-test-slave-kghz"
server_key: "a67348942f8b1cc0405be056d5b616a0"
server_key_iv: "1b7ca834461565d92f3c857721ef0a7c"
server_key_version: "encryptionkey@0"
I20260925 04:48:28.836278 20865 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 04:48:28.839006 20880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:48:28.840572 20865 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:48:28.840735 20865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790311449873301-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790311449873301-393-0/minicluster-data/master-0/wal
uuid: "b4907016da864d88911e1749421566b4"
format_stamp: "Formatted at 2026-09-25 04:48:28 on dist-test-slave-kghz"
server_key: "a67348942f8b1cc0405be056d5b616a0"
server_key_iv: "1b7ca834461565d92f3c857721ef0a7c"
server_key_version: "encryptionkey@0"
I20260925 04:48:28.840885 20865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790311449873301-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790311449873301-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790311449873301-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:48:28.845543 20865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:48:28.848809 20865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:48:28.849015 20865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:48:28.855854 20865 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34691
I20260925 04:48:28.855916 20932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34691 every 8 connection(s)
I20260925 04:48:28.856805 20865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
I20260925 04:48:28.859668 20933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:48:28.865033 20933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4907016da864d88911e1749421566b4: Bootstrap starting.
I20260925 04:48:28.866503 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 20865
I20260925 04:48:28.866636 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790311449873301-393-0/minicluster-data/master-0/wal/instance
I20260925 04:48:28.866940 393 external_mini_cluster.cc:1428] Setting key 8c5962be05a136ea6a71ca7cff9c3c8a
I20260925 04:48:28.867861 20933 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4907016da864d88911e1749421566b4: Neither blocks nor log segments found. Creating new log.
I20260925 04:48:28.868705 20933 log.cc:824] T 00000000000000000000000000000000 P b4907016da864d88911e1749421566b4: Log is configured to *not* fsync() on all Append() calls
I20260925 04:48:28.871097 20933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4907016da864d88911e1749421566b4: No bootstrap required, opened a new log
I20260925 04:48:28.872195 20936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:28.868074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50381 (local address 127.0.0.1:34691)
0925 04:48:28.868532 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:28.868546 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:28.868581 (+ 35us) server_negotiation.cc:407] Connection header received
0925 04:48:28.869361 (+ 780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:28.869396 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:28.869694 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:28.869982 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:28.870944 (+ 962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:28.870949 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:28.870960 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:48:28.870970 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:28.871021 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:48:28.871039 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:28.871128 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:28.871315 (+ 187us) server_negotiation.cc:299] Negotiation successful
0925 04:48:28.871510 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":139,"threads_started":1}
I20260925 04:48:28.875253 393 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:48:28.875250 20933 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4907016da864d88911e1749421566b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4907016da864d88911e1749421566b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34691 } }
I20260925 04:48:28.875530 20933 raft_consensus.cc:374] T 00000000000000000000000000000000 P b4907016da864d88911e1749421566b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:48:28.875602 20933 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4907016da864d88911e1749421566b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4907016da864d88911e1749421566b4, State: Initialized, Role: FOLLOWER
I20260925 04:48:28.876052 20933 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4907016da864d88911e1749421566b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4907016da864d88911e1749421566b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34691 } }
I20260925 04:48:28.876180 20933 raft_consensus.cc:388] T 00000000000000000000000000000000 P b4907016da864d88911e1749421566b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:48:28.876291 20933 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4907016da864d88911e1749421566b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:48:28.876415 20933 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4907016da864d88911e1749421566b4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:48:28.877401 20933 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4907016da864d88911e1749421566b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4907016da864d88911e1749421566b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34691 } }
I20260925 04:48:28.877820 20933 leader_election.cc:302] T 00000000000000000000000000000000 P b4907016da864d88911e1749421566b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4907016da864d88911e1749421566b4; no voters:
I20260925 04:48:28.878227 20933 leader_election.cc:288] T 00000000000000000000000000000000 P b4907016da864d88911e1749421566b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:48:28.878369 20943 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4907016da864d88911e1749421566b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:48:28.878616 20943 raft_consensus.cc:686] T 00000000000000000000000000000000 P b4907016da864d88911e1749421566b4 [term 1 LEADER]: Becoming Leader. State: Replica: b4907016da864d88911e1749421566b4, State: Running, Role: LEADER
I20260925 04:48:28.878997 20943 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4907016da864d88911e1749421566b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4907016da864d88911e1749421566b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34691 } }
I20260925 04:48:28.879552 20936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:28.877629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35839 (local address 127.0.0.1:34691)
0925 04:48:28.877828 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:28.877831 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:28.877919 (+ 88us) server_negotiation.cc:407] Connection header received
0925 04:48:28.878057 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:28.878059 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:28.878090 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:28.878151 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:28.879006 (+ 855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:28.879010 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:28.879013 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:48:28.879023 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:28.879050 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:48:28.879064 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:28.879140 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:28.879403 (+ 263us) server_negotiation.cc:299] Negotiation successful
0925 04:48:28.879439 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260925 04:48:28.879653 20933 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4907016da864d88911e1749421566b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:48:28.880774 20944 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4907016da864d88911e1749421566b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b4907016da864d88911e1749421566b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4907016da864d88911e1749421566b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34691 } } }
I20260925 04:48:28.880903 20944 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4907016da864d88911e1749421566b4 [sys.catalog]: This master's current role is: LEADER
I20260925 04:48:28.881299 20952 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:48:28.881273 20945 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4907016da864d88911e1749421566b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4907016da864d88911e1749421566b4. Latest consensus state: current_term: 1 leader_uuid: "b4907016da864d88911e1749421566b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4907016da864d88911e1749421566b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34691 } } }
I20260925 04:48:28.881448 20945 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4907016da864d88911e1749421566b4 [sys.catalog]: This master's current role is: LEADER
I20260925 04:48:28.883978 20952 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:48:28.889583 20952 catalog_manager.cc:1347] Generated new cluster ID: 014b4c9473d94aaba3441e2721c0e1d0
I20260925 04:48:28.889655 20952 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:48:28.918031 20952 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:48:28.918431 20952 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:48:28.925909 20952 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b4907016da864d88911e1749421566b4: Generated new TSK 0
I20260925 04:48:28.926646 20952 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:48:28.934952 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 20865
2026-09-25T04:48:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1425 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790311449873301-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:48:28 dist-test-slave-kghz krb5kdc[20964](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:48:28 dist-test-slave-kghz krb5kdc[20964](info): set up 2 sockets
Sep 25 04:48:28 dist-test-slave-kghz krb5kdc[20964](info): commencing operation
krb5kdc: starting...
W20260925 04:48:29.974948 393 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790311449873301-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790311449873301-393-0/krb5kdc/test-user.keytab.
Sep 25 04:48:30 dist-test-slave-kghz krb5kdc[20964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311710, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:48:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:48:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790311449873301-393-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790311449873301-393-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 04:48:30.103775 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790311449873301-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790311449873301-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790311449873301-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33045
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790311449873301-393-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33045
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:48:30.217010 20977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:48:30.217276 20977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:48:30.217319 20977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:48:30.221344 20977 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:48:30.221410 20977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:48:30.221428 20977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:48:30.221446 20977 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:48:30.221460 20977 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:48:30.226718 20977 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790311449873301-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790311449873301-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33045
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790311449873301-393-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790311449873301-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:48:30.227728 20977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:48:30.228711 20977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:48:30.237936 20985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:48:30.237949 20982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:48:30.237890 20983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:48:30.237919 20977 server_base.cc:1034] running on GCE node
I20260925 04:48:30.238796 20977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:48:30.240010 20977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:48:30.241189 20977 hybrid_clock.cc:648] HybridClock initialized: now 1790311710241153 us; error 64 us; skew 500 ppm
Sep 25 04:48:30 dist-test-slave-kghz krb5kdc[20964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311710, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:48:30.249081 20977 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 04:48:30.250144 20977 webserver.cc:460] Webserver started at http://127.0.0.1:38609/ using document root <none> and password file <none>
I20260925 04:48:30.250797 20977 fs_manager.cc:359] Metadata directory not provided
I20260925 04:48:30.250905 20977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:48:30.251158 20977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:48:30.253499 20977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790311449873301-393-0/minicluster-data/master-0/data/instance:
uuid: "f09aac73864a4caa92dc0d0e51e6f77a"
format_stamp: "Formatted at 2026-09-25 04:48:30 on dist-test-slave-kghz"
server_key: "6e94860acdb8da0491141172f6dae765"
server_key_iv: "8736526bc9d6dad8b8341c60ec70349d"
server_key_version: "encryptionkey@0"
I20260925 04:48:30.254109 20977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790311449873301-393-0/minicluster-data/master-0/wal/instance:
uuid: "f09aac73864a4caa92dc0d0e51e6f77a"
format_stamp: "Formatted at 2026-09-25 04:48:30 on dist-test-slave-kghz"
server_key: "6e94860acdb8da0491141172f6dae765"
server_key_iv: "8736526bc9d6dad8b8341c60ec70349d"
server_key_version: "encryptionkey@0"
I20260925 04:48:30.258353 20977 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 04:48:30.261041 20992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:48:30.262650 20977 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:48:30.262804 20977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790311449873301-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790311449873301-393-0/minicluster-data/master-0/wal
uuid: "f09aac73864a4caa92dc0d0e51e6f77a"
format_stamp: "Formatted at 2026-09-25 04:48:30 on dist-test-slave-kghz"
server_key: "6e94860acdb8da0491141172f6dae765"
server_key_iv: "8736526bc9d6dad8b8341c60ec70349d"
server_key_version: "encryptionkey@0"
I20260925 04:48:30.262952 20977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790311449873301-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790311449873301-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790311449873301-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:48:30.288434 20977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:48:30.291973 20977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:48:30.292212 20977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:48:30.299532 20977 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33045
I20260925 04:48:30.299547 21044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33045 every 8 connection(s)
I20260925 04:48:30.300567 20977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
I20260925 04:48:30.303398 21045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:48:30.308821 21045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f09aac73864a4caa92dc0d0e51e6f77a: Bootstrap starting.
I20260925 04:48:30.310823 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 20977
I20260925 04:48:30.310984 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790311449873301-393-0/minicluster-data/master-0/wal/instance
I20260925 04:48:30.311287 393 external_mini_cluster.cc:1428] Setting key 44beac20e792f02ebb3e3b58dcf0cd4f
I20260925 04:48:30.311518 21045 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f09aac73864a4caa92dc0d0e51e6f77a: Neither blocks nor log segments found. Creating new log.
I20260925 04:48:30.312536 21045 log.cc:824] T 00000000000000000000000000000000 P f09aac73864a4caa92dc0d0e51e6f77a: Log is configured to *not* fsync() on all Append() calls
I20260925 04:48:30.315109 21045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f09aac73864a4caa92dc0d0e51e6f77a: No bootstrap required, opened a new log
I20260925 04:48:30.319630 21045 raft_consensus.cc:348] T 00000000000000000000000000000000 P f09aac73864a4caa92dc0d0e51e6f77a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f09aac73864a4caa92dc0d0e51e6f77a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33045 } }
I20260925 04:48:30.319954 21045 raft_consensus.cc:374] T 00000000000000000000000000000000 P f09aac73864a4caa92dc0d0e51e6f77a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:48:30.320035 21045 raft_consensus.cc:729] T 00000000000000000000000000000000 P f09aac73864a4caa92dc0d0e51e6f77a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f09aac73864a4caa92dc0d0e51e6f77a, State: Initialized, Role: FOLLOWER
I20260925 04:48:30.320492 21045 consensus_queue.cc:260] T 00000000000000000000000000000000 P f09aac73864a4caa92dc0d0e51e6f77a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f09aac73864a4caa92dc0d0e51e6f77a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33045 } }
I20260925 04:48:30.320616 21045 raft_consensus.cc:388] T 00000000000000000000000000000000 P f09aac73864a4caa92dc0d0e51e6f77a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:48:30.320705 21045 raft_consensus.cc:482] T 00000000000000000000000000000000 P f09aac73864a4caa92dc0d0e51e6f77a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:48:30.320842 21045 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f09aac73864a4caa92dc0d0e51e6f77a [term 0 FOLLOWER]: Advancing to term 1
Sep 25 04:48:30 dist-test-slave-kghz krb5kdc[20964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311710, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 04:48:30.322094 21045 raft_consensus.cc:504] T 00000000000000000000000000000000 P f09aac73864a4caa92dc0d0e51e6f77a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f09aac73864a4caa92dc0d0e51e6f77a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33045 } }
I20260925 04:48:30.322467 21045 leader_election.cc:302] T 00000000000000000000000000000000 P f09aac73864a4caa92dc0d0e51e6f77a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f09aac73864a4caa92dc0d0e51e6f77a; no voters:
I20260925 04:48:30.322854 21045 leader_election.cc:288] T 00000000000000000000000000000000 P f09aac73864a4caa92dc0d0e51e6f77a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:48:30.323035 21050 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f09aac73864a4caa92dc0d0e51e6f77a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:48:30.323278 21050 raft_consensus.cc:686] T 00000000000000000000000000000000 P f09aac73864a4caa92dc0d0e51e6f77a [term 1 LEADER]: Becoming Leader. State: Replica: f09aac73864a4caa92dc0d0e51e6f77a, State: Running, Role: LEADER
I20260925 04:48:30.323642 21050 consensus_queue.cc:237] T 00000000000000000000000000000000 P f09aac73864a4caa92dc0d0e51e6f77a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f09aac73864a4caa92dc0d0e51e6f77a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33045 } }
I20260925 04:48:30.324337 21045 sys_catalog.cc:564] T 00000000000000000000000000000000 P f09aac73864a4caa92dc0d0e51e6f77a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:48:30.325394 21052 sys_catalog.cc:455] T 00000000000000000000000000000000 P f09aac73864a4caa92dc0d0e51e6f77a [sys.catalog]: SysCatalogTable state changed. Reason: New leader f09aac73864a4caa92dc0d0e51e6f77a. Latest consensus state: current_term: 1 leader_uuid: "f09aac73864a4caa92dc0d0e51e6f77a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f09aac73864a4caa92dc0d0e51e6f77a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33045 } } }
I20260925 04:48:30.325532 21052 sys_catalog.cc:458] T 00000000000000000000000000000000 P f09aac73864a4caa92dc0d0e51e6f77a [sys.catalog]: This master's current role is: LEADER
I20260925 04:48:30.325599 21051 sys_catalog.cc:455] T 00000000000000000000000000000000 P f09aac73864a4caa92dc0d0e51e6f77a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f09aac73864a4caa92dc0d0e51e6f77a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f09aac73864a4caa92dc0d0e51e6f77a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33045 } } }
I20260925 04:48:30.325670 21051 sys_catalog.cc:458] T 00000000000000000000000000000000 P f09aac73864a4caa92dc0d0e51e6f77a [sys.catalog]: This master's current role is: LEADER
I20260925 04:48:30.325989 21059 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:48:30.328553 21049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:30.312694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42261 (local address 127.0.0.1:33045)
0925 04:48:30.313181 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:30.313199 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:30.313237 (+ 38us) server_negotiation.cc:407] Connection header received
0925 04:48:30.314162 (+ 925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:30.314189 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:30.314672 (+ 483us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:30.315059 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:30.315935 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:30.317490 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:30.318521 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:30.319242 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:30.319413 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:30.322610 (+ 3197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:30.322616 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:30.322634 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:48:30.322666 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:30.324829 (+ 2163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:30.325511 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:30.325517 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:30.325523 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:30.325690 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:30.326025 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:30.326031 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:30.326036 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:30.326669 (+ 633us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:30.326881 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:30.327262 (+ 381us) server_negotiation.cc:299] Negotiation successful
0925 04:48:30.327595 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"spinlock_wait_cycles":36608,"thread_start_us":169,"threads_started":1}
I20260925 04:48:30.329137 21059 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:48:30.331555 393 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:48:30.335485 21059 catalog_manager.cc:1347] Generated new cluster ID: 5c37858094504e7ca774e5f64178bb0a
I20260925 04:48:30.335541 21059 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:48:30.342808 21049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:30.333956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43781 (local address 127.0.0.1:33045)
0925 04:48:30.334101 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:30.334107 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:30.334126 (+ 19us) server_negotiation.cc:407] Connection header received
0925 04:48:30.334172 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:30.334176 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:30.334231 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:30.334371 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:30.335142 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:30.335982 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:30.337165 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:30.337518 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:30.337612 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:30.339256 (+ 1644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:30.339260 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:30.339263 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:48:30.339286 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:30.341058 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:30.341576 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:30.341579 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:30.341581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:30.341660 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:30.341937 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:30.341940 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:30.341942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:30.342129 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:30.342265 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:30.342528 (+ 263us) server_negotiation.cc:299] Negotiation successful
0925 04:48:30.342638 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 04:48:30.345158 21059 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:48:30.346417 21059 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:48:30.357153 21059 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f09aac73864a4caa92dc0d0e51e6f77a: Generated new TSK 0
I20260925 04:48:30.358073 21059 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:48:30.362077 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 20977
2026-09-25T04:48:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1427 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790311449873301-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:48:30 dist-test-slave-kghz krb5kdc[21076](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:48:30 dist-test-slave-kghz krb5kdc[21076](info): set up 2 sockets
Sep 25 04:48:30 dist-test-slave-kghz krb5kdc[21076](info): commencing operation
krb5kdc: starting...
W20260925 04:48:31.398625 393 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790311449873301-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790311449873301-393-0/krb5kdc/test-user.keytab.
Sep 25 04:48:31 dist-test-slave-kghz krb5kdc[21076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:48:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:48:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790311449873301-393-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790311449873301-393-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 04:48:31.529450 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790311449873301-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790311449873301-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790311449873301-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36029
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790311449873301-393-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36029
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:48:31.642060 21089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:48:31.642423 21089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:48:31.642509 21089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:48:31.646664 21089 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:48:31.646767 21089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:48:31.646808 21089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:48:31.646876 21089 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:48:31.646916 21089 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:48:31.652362 21089 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790311449873301-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790311449873301-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36029
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790311449873301-393-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21089
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790311449873301-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:48:31.653493 21089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:48:31.654619 21089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:48:31.663399 21095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:48:31.663427 21097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:48:31.663511 21089 server_base.cc:1034] running on GCE node
W20260925 04:48:31.663697 21094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:48:31.664278 21089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:48:31.665433 21089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:48:31.666627 21089 hybrid_clock.cc:648] HybridClock initialized: now 1790311711666609 us; error 27 us; skew 500 ppm
Sep 25 04:48:31 dist-test-slave-kghz krb5kdc[21076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311711, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:48:31.669314 21089 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 04:48:31.670177 21089 webserver.cc:460] Webserver started at http://127.0.0.1:39567/ using document root <none> and password file <none>
I20260925 04:48:31.670758 21089 fs_manager.cc:359] Metadata directory not provided
I20260925 04:48:31.670819 21089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:48:31.670987 21089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:48:31.673017 21089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790311449873301-393-0/minicluster-data/master-0/data/instance:
uuid: "f4a58e230ddf4e15b5565b879d2d377f"
format_stamp: "Formatted at 2026-09-25 04:48:31 on dist-test-slave-kghz"
server_key: "10fc2d2b467c054247954d17faec15cf"
server_key_iv: "bda5bfe12e006ae3a615ef977f6d0380"
server_key_version: "encryptionkey@0"
I20260925 04:48:31.673559 21089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790311449873301-393-0/minicluster-data/master-0/wal/instance:
uuid: "f4a58e230ddf4e15b5565b879d2d377f"
format_stamp: "Formatted at 2026-09-25 04:48:31 on dist-test-slave-kghz"
server_key: "10fc2d2b467c054247954d17faec15cf"
server_key_iv: "bda5bfe12e006ae3a615ef977f6d0380"
server_key_version: "encryptionkey@0"
I20260925 04:48:31.677644 21089 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 04:48:31.680323 21104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:48:31.681682 21089 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 04:48:31.681891 21089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790311449873301-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790311449873301-393-0/minicluster-data/master-0/wal
uuid: "f4a58e230ddf4e15b5565b879d2d377f"
format_stamp: "Formatted at 2026-09-25 04:48:31 on dist-test-slave-kghz"
server_key: "10fc2d2b467c054247954d17faec15cf"
server_key_iv: "bda5bfe12e006ae3a615ef977f6d0380"
server_key_version: "encryptionkey@0"
I20260925 04:48:31.682024 21089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790311449873301-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790311449873301-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790311449873301-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:48:31.695539 21089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:48:31.699007 21089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:48:31.699221 21089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:48:31.706382 21089 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36029
I20260925 04:48:31.706415 21156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36029 every 8 connection(s)
I20260925 04:48:31.707386 21089 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
I20260925 04:48:31.710237 21157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:48:31.715818 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 21089
I20260925 04:48:31.715953 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790311449873301-393-0/minicluster-data/master-0/wal/instance
I20260925 04:48:31.716148 21157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4a58e230ddf4e15b5565b879d2d377f: Bootstrap starting.
I20260925 04:48:31.716312 393 external_mini_cluster.cc:1428] Setting key 3ad607016c562f686dbf673dd0c63fe5
I20260925 04:48:31.718930 21157 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4a58e230ddf4e15b5565b879d2d377f: Neither blocks nor log segments found. Creating new log.
I20260925 04:48:31.719834 21157 log.cc:824] T 00000000000000000000000000000000 P f4a58e230ddf4e15b5565b879d2d377f: Log is configured to *not* fsync() on all Append() calls
I20260925 04:48:31.722007 21157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4a58e230ddf4e15b5565b879d2d377f: No bootstrap required, opened a new log
I20260925 04:48:31.725615 21160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:31.717823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34161 (local address 127.0.0.1:36029)
0925 04:48:31.718302 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:31.718325 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:31.718361 (+ 36us) server_negotiation.cc:407] Connection header received
0925 04:48:31.719034 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:31.719082 (+ 48us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:31.719474 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:31.719783 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:31.720171 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:31.721113 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:31.722330 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:31.722805 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:31.722928 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:31.723866 (+ 938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:31.723871 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:31.723883 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:48:31.724289 (+ 406us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:31.724346 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:48:31.724368 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:31.724450 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:31.724622 (+ 172us) server_negotiation.cc:299] Negotiation successful
0925 04:48:31.724894 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":147,"threads_started":1}
I20260925 04:48:31.726183 21157 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4a58e230ddf4e15b5565b879d2d377f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4a58e230ddf4e15b5565b879d2d377f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36029 } }
I20260925 04:48:31.726487 21157 raft_consensus.cc:374] T 00000000000000000000000000000000 P f4a58e230ddf4e15b5565b879d2d377f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:48:31.726567 21157 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4a58e230ddf4e15b5565b879d2d377f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4a58e230ddf4e15b5565b879d2d377f, State: Initialized, Role: FOLLOWER
I20260925 04:48:31.727048 21157 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4a58e230ddf4e15b5565b879d2d377f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4a58e230ddf4e15b5565b879d2d377f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36029 } }
I20260925 04:48:31.727202 21157 raft_consensus.cc:388] T 00000000000000000000000000000000 P f4a58e230ddf4e15b5565b879d2d377f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:48:31.727265 21157 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4a58e230ddf4e15b5565b879d2d377f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:48:31.727368 21157 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4a58e230ddf4e15b5565b879d2d377f [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:48:31.728538 21157 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4a58e230ddf4e15b5565b879d2d377f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4a58e230ddf4e15b5565b879d2d377f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36029 } }
I20260925 04:48:31.728703 393 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:48:31.728904 21157 leader_election.cc:302] T 00000000000000000000000000000000 P f4a58e230ddf4e15b5565b879d2d377f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4a58e230ddf4e15b5565b879d2d377f; no voters:
I20260925 04:48:31.729223 21157 leader_election.cc:288] T 00000000000000000000000000000000 P f4a58e230ddf4e15b5565b879d2d377f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:48:31.729342 21162 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4a58e230ddf4e15b5565b879d2d377f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:48:31.729554 21162 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4a58e230ddf4e15b5565b879d2d377f [term 1 LEADER]: Becoming Leader. State: Replica: f4a58e230ddf4e15b5565b879d2d377f, State: Running, Role: LEADER
I20260925 04:48:31.729928 21162 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4a58e230ddf4e15b5565b879d2d377f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4a58e230ddf4e15b5565b879d2d377f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36029 } }
I20260925 04:48:31.730528 21157 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4a58e230ddf4e15b5565b879d2d377f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:48:31.731455 21162 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4a58e230ddf4e15b5565b879d2d377f [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4a58e230ddf4e15b5565b879d2d377f. Latest consensus state: current_term: 1 leader_uuid: "f4a58e230ddf4e15b5565b879d2d377f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4a58e230ddf4e15b5565b879d2d377f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36029 } } }
I20260925 04:48:31.731624 21162 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4a58e230ddf4e15b5565b879d2d377f [sys.catalog]: This master's current role is: LEADER
I20260925 04:48:31.731906 21163 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4a58e230ddf4e15b5565b879d2d377f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f4a58e230ddf4e15b5565b879d2d377f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4a58e230ddf4e15b5565b879d2d377f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36029 } } }
I20260925 04:48:31.732002 21163 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4a58e230ddf4e15b5565b879d2d377f [sys.catalog]: This master's current role is: LEADER
I20260925 04:48:31.732040 21176 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:48:31.735186 21176 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:48:31.737576 21160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:31.730989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42075 (local address 127.0.0.1:36029)
0925 04:48:31.731108 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:31.731113 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:31.733016 (+ 1903us) server_negotiation.cc:407] Connection header received
0925 04:48:31.733202 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:31.733206 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:31.733263 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:31.733358 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:31.733649 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:31.734418 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:31.735572 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:31.735923 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:31.736000 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:31.737082 (+ 1082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:31.737086 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:31.737089 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:48:31.737101 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:31.737129 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:48:31.737141 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:31.737219 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:31.737370 (+ 151us) server_negotiation.cc:299] Negotiation successful
0925 04:48:31.737435 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 04:48:31.741094 21176 catalog_manager.cc:1347] Generated new cluster ID: 14eea01d345d4639bd8d77be68c87a82
I20260925 04:48:31.741163 21176 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:48:31.750545 21176 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:48:31.751502 21176 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:48:31.764627 21176 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f4a58e230ddf4e15b5565b879d2d377f: Generated new TSK 0
I20260925 04:48:31.765625 21176 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:48:31.776870 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 21089
2026-09-25T04:48:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1414 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790311449873301-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:48:31 dist-test-slave-kghz krb5kdc[21188](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:48:31 dist-test-slave-kghz krb5kdc[21188](info): set up 2 sockets
Sep 25 04:48:31 dist-test-slave-kghz krb5kdc[21188](info): commencing operation
krb5kdc: starting...
W20260925 04:48:32.822487 393 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790311449873301-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790311449873301-393-0/krb5kdc/test-user.keytab.
Sep 25 04:48:32 dist-test-slave-kghz krb5kdc[21188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311712, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:48:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:48:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790311449873301-393-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790311449873301-393-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 04:48:32.954797 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790311449873301-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790311449873301-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790311449873301-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37249
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790311449873301-393-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37249
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:48:33.067078 21201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:48:33.067365 21201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:48:33.067411 21201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:48:33.071497 21201 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:48:33.071563 21201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:48:33.071581 21201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:48:33.071599 21201 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:48:33.071614 21201 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:48:33.077032 21201 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790311449873301-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790311449873301-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37249
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790311449873301-393-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790311449873301-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:48:33.078235 21201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:48:33.079387 21201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:48:33.088310 21209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:48:33.088423 21206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:48:33.088500 21201 server_base.cc:1034] running on GCE node
W20260925 04:48:33.088413 21207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:48:33.089181 21201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:48:33.090399 21201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:48:33.091579 21201 hybrid_clock.cc:648] HybridClock initialized: now 1790311713091546 us; error 45 us; skew 500 ppm
Sep 25 04:48:33 dist-test-slave-kghz krb5kdc[21188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311713, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:48:33.094493 21201 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 04:48:33.095353 21201 webserver.cc:460] Webserver started at http://127.0.0.1:41503/ using document root <none> and password file <none>
I20260925 04:48:33.095933 21201 fs_manager.cc:359] Metadata directory not provided
I20260925 04:48:33.095999 21201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:48:33.096170 21201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:48:33.098320 21201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790311449873301-393-0/minicluster-data/master-0/data/instance:
uuid: "d7797ea12c574a45a5a769f2943babda"
format_stamp: "Formatted at 2026-09-25 04:48:33 on dist-test-slave-kghz"
server_key: "3b02b25f49e27451a884e0815dd24630"
server_key_iv: "29dfc48bf4a7fec80ccbed2caf8d2aca"
server_key_version: "encryptionkey@0"
I20260925 04:48:33.098850 21201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790311449873301-393-0/minicluster-data/master-0/wal/instance:
uuid: "d7797ea12c574a45a5a769f2943babda"
format_stamp: "Formatted at 2026-09-25 04:48:33 on dist-test-slave-kghz"
server_key: "3b02b25f49e27451a884e0815dd24630"
server_key_iv: "29dfc48bf4a7fec80ccbed2caf8d2aca"
server_key_version: "encryptionkey@0"
I20260925 04:48:33.102596 21201 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 04:48:33.105044 21216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:48:33.106443 21201 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 04:48:33.106585 21201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790311449873301-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790311449873301-393-0/minicluster-data/master-0/wal
uuid: "d7797ea12c574a45a5a769f2943babda"
format_stamp: "Formatted at 2026-09-25 04:48:33 on dist-test-slave-kghz"
server_key: "3b02b25f49e27451a884e0815dd24630"
server_key_iv: "29dfc48bf4a7fec80ccbed2caf8d2aca"
server_key_version: "encryptionkey@0"
I20260925 04:48:33.106700 21201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790311449873301-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790311449873301-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790311449873301-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:48:33.111439 21201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:48:33.114918 21201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:48:33.115182 21201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:48:33.122733 21201 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37249
I20260925 04:48:33.122756 21268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37249 every 8 connection(s)
I20260925 04:48:33.123814 21201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
I20260925 04:48:33.126830 21269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:48:33.131556 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 21201
I20260925 04:48:33.131713 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790311449873301-393-0/minicluster-data/master-0/wal/instance
I20260925 04:48:33.132043 393 external_mini_cluster.cc:1428] Setting key 1128987563c85e7b82aecaab77f86c1a
I20260925 04:48:33.132845 21269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7797ea12c574a45a5a769f2943babda: Bootstrap starting.
I20260925 04:48:33.135346 21269 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d7797ea12c574a45a5a769f2943babda: Neither blocks nor log segments found. Creating new log.
I20260925 04:48:33.136266 21269 log.cc:824] T 00000000000000000000000000000000 P d7797ea12c574a45a5a769f2943babda: Log is configured to *not* fsync() on all Append() calls
I20260925 04:48:33.137768 21272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:33.133540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52601 (local address 127.0.0.1:37249)
0925 04:48:33.134001 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:33.134016 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:33.134058 (+ 42us) server_negotiation.cc:407] Connection header received
0925 04:48:33.134781 (+ 723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:33.134800 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:33.135102 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:33.135477 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:33.136281 (+ 804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:33.136285 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:33.136298 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:48:33.136309 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:33.136362 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:48:33.136382 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:33.136518 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:33.136766 (+ 248us) server_negotiation.cc:299] Negotiation successful
0925 04:48:33.136962 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":137,"threads_started":1}
I20260925 04:48:33.138655 21269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7797ea12c574a45a5a769f2943babda: No bootstrap required, opened a new log
I20260925 04:48:33.140861 393 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:48:33.142797 21269 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7797ea12c574a45a5a769f2943babda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7797ea12c574a45a5a769f2943babda" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37249 } }
I20260925 04:48:33.143270 21269 raft_consensus.cc:374] T 00000000000000000000000000000000 P d7797ea12c574a45a5a769f2943babda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:48:33.143355 21269 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7797ea12c574a45a5a769f2943babda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7797ea12c574a45a5a769f2943babda, State: Initialized, Role: FOLLOWER
I20260925 04:48:33.143862 21269 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7797ea12c574a45a5a769f2943babda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7797ea12c574a45a5a769f2943babda" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37249 } }
I20260925 04:48:33.144061 21269 raft_consensus.cc:388] T 00000000000000000000000000000000 P d7797ea12c574a45a5a769f2943babda [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:48:33.144135 21269 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7797ea12c574a45a5a769f2943babda [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:48:33.144248 21269 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7797ea12c574a45a5a769f2943babda [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:48:33.144850 21272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:33.143045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34647 (local address 127.0.0.1:37249)
0925 04:48:33.143208 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:33.143213 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:33.143310 (+ 97us) server_negotiation.cc:407] Connection header received
0925 04:48:33.143525 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:33.143528 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:33.143567 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:33.143655 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:33.144455 (+ 800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:33.144460 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:33.144463 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:48:33.144476 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:33.144505 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:48:33.144517 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:33.144616 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:33.144689 (+ 73us) server_negotiation.cc:299] Negotiation successful
0925 04:48:33.144729 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 04:48:33.145453 21269 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7797ea12c574a45a5a769f2943babda [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7797ea12c574a45a5a769f2943babda" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37249 } }
I20260925 04:48:33.145846 21269 leader_election.cc:302] T 00000000000000000000000000000000 P d7797ea12c574a45a5a769f2943babda [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d7797ea12c574a45a5a769f2943babda; no voters:
I20260925 04:48:33.146179 21269 leader_election.cc:288] T 00000000000000000000000000000000 P d7797ea12c574a45a5a769f2943babda [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:48:33.146346 21279 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7797ea12c574a45a5a769f2943babda [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:48:33.146574 21279 raft_consensus.cc:686] T 00000000000000000000000000000000 P d7797ea12c574a45a5a769f2943babda [term 1 LEADER]: Becoming Leader. State: Replica: d7797ea12c574a45a5a769f2943babda, State: Running, Role: LEADER
I20260925 04:48:33.146905 21279 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7797ea12c574a45a5a769f2943babda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7797ea12c574a45a5a769f2943babda" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37249 } }
I20260925 04:48:33.147442 21269 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7797ea12c574a45a5a769f2943babda [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:48:33.148646 21280 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7797ea12c574a45a5a769f2943babda [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d7797ea12c574a45a5a769f2943babda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7797ea12c574a45a5a769f2943babda" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37249 } } }
I20260925 04:48:33.148602 21281 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7797ea12c574a45a5a769f2943babda [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7797ea12c574a45a5a769f2943babda. Latest consensus state: current_term: 1 leader_uuid: "d7797ea12c574a45a5a769f2943babda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7797ea12c574a45a5a769f2943babda" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37249 } } }
I20260925 04:48:33.148741 21280 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7797ea12c574a45a5a769f2943babda [sys.catalog]: This master's current role is: LEADER
I20260925 04:48:33.148741 21281 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7797ea12c574a45a5a769f2943babda [sys.catalog]: This master's current role is: LEADER
I20260925 04:48:33.149173 21288 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:48:33.152344 21288 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:48:33.157945 21288 catalog_manager.cc:1347] Generated new cluster ID: 29f977bc733a4d6181832e1d9bc94d90
I20260925 04:48:33.158022 21288 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:48:33.184775 21288 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:48:33.185158 21288 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:48:33.192673 21288 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d7797ea12c574a45a5a769f2943babda: Generated new TSK 0
I20260925 04:48:33.193528 21288 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:48:33.208774 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 21201
2026-09-25T04:48:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1432 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790311449873301-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:48:33 dist-test-slave-kghz krb5kdc[21300](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:48:33 dist-test-slave-kghz krb5kdc[21300](info): set up 2 sockets
Sep 25 04:48:33 dist-test-slave-kghz krb5kdc[21300](info): commencing operation
krb5kdc: starting...
W20260925 04:48:34.262619 393 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790311449873301-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790311449873301-393-0/krb5kdc/test-user.keytab.
Sep 25 04:48:34 dist-test-slave-kghz krb5kdc[21300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:48:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:48:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790311449873301-393-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790311449873301-393-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 04:48:34.391105 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790311449873301-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790311449873301-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790311449873301-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35829
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790311449873301-393-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35829
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:48:34.502736 21313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:48:34.503075 21313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:48:34.503159 21313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:48:34.507288 21313 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:48:34.507383 21313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:48:34.507417 21313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:48:34.507445 21313 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:48:34.507477 21313 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:48:34.512952 21313 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790311449873301-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790311449873301-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790311449873301-393-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790311449873301-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:48:34.514103 21313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:48:34.515213 21313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:48:34.523931 21319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:48:34.524000 21313 server_base.cc:1034] running on GCE node
W20260925 04:48:34.523981 21321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:48:34.523940 21318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:48:34.524654 21313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:48:34.525738 21313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:48:34.526945 21313 hybrid_clock.cc:648] HybridClock initialized: now 1790311714526927 us; error 30 us; skew 500 ppm
Sep 25 04:48:34 dist-test-slave-kghz krb5kdc[21300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311714, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:48:34.529671 21313 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 04:48:34.530659 21313 webserver.cc:460] Webserver started at http://127.0.0.1:44441/ using document root <none> and password file <none>
I20260925 04:48:34.531219 21313 fs_manager.cc:359] Metadata directory not provided
I20260925 04:48:34.531309 21313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:48:34.531541 21313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:48:34.533527 21313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790311449873301-393-0/minicluster-data/master-0/data/instance:
uuid: "93679fe0417c465787479054a42cb1ce"
format_stamp: "Formatted at 2026-09-25 04:48:34 on dist-test-slave-kghz"
server_key: "0d424a6a5c75e2975e523a1bdf441221"
server_key_iv: "6fc425c7d93f83c004bcfd7fe591401f"
server_key_version: "encryptionkey@0"
I20260925 04:48:34.534094 21313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790311449873301-393-0/minicluster-data/master-0/wal/instance:
uuid: "93679fe0417c465787479054a42cb1ce"
format_stamp: "Formatted at 2026-09-25 04:48:34 on dist-test-slave-kghz"
server_key: "0d424a6a5c75e2975e523a1bdf441221"
server_key_iv: "6fc425c7d93f83c004bcfd7fe591401f"
server_key_version: "encryptionkey@0"
I20260925 04:48:34.537801 21313 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 04:48:34.540216 21328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:48:34.541498 21313 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 04:48:34.541639 21313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790311449873301-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790311449873301-393-0/minicluster-data/master-0/wal
uuid: "93679fe0417c465787479054a42cb1ce"
format_stamp: "Formatted at 2026-09-25 04:48:34 on dist-test-slave-kghz"
server_key: "0d424a6a5c75e2975e523a1bdf441221"
server_key_iv: "6fc425c7d93f83c004bcfd7fe591401f"
server_key_version: "encryptionkey@0"
I20260925 04:48:34.541755 21313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790311449873301-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790311449873301-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790311449873301-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:48:34.563768 21313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:48:34.566684 21313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:48:34.566929 21313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:48:34.574149 21313 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35829
I20260925 04:48:34.574174 21380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35829 every 8 connection(s)
I20260925 04:48:34.575124 21313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
I20260925 04:48:34.577822 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 21313
I20260925 04:48:34.577860 21381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:48:34.577977 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790311449873301-393-0/minicluster-data/master-0/wal/instance
I20260925 04:48:34.578322 393 external_mini_cluster.cc:1428] Setting key 27686040765fc8bd74781031f56e380b
I20260925 04:48:34.584122 21381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93679fe0417c465787479054a42cb1ce: Bootstrap starting.
I20260925 04:48:34.586489 21381 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93679fe0417c465787479054a42cb1ce: Neither blocks nor log segments found. Creating new log.
Sep 25 04:48:34 dist-test-slave-kghz krb5kdc[21300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 04:48:34.587442 21381 log.cc:824] T 00000000000000000000000000000000 P 93679fe0417c465787479054a42cb1ce: Log is configured to *not* fsync() on all Append() calls
I20260925 04:48:34.589877 21381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93679fe0417c465787479054a42cb1ce: No bootstrap required, opened a new log
I20260925 04:48:34.593394 21384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:34.579729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.103:45587 (local address 127.0.0.1:35829)
0925 04:48:34.580182 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:34.580197 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:34.580231 (+ 34us) server_negotiation.cc:407] Connection header received
0925 04:48:34.581007 (+ 776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:34.581027 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:34.581369 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:34.581716 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:34.582557 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:34.583523 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:34.584427 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:34.584896 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:34.585062 (+ 166us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:34.588047 (+ 2985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:34.588053 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:34.588072 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:48:34.588103 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:34.590049 (+ 1946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:34.590680 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:34.590685 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:34.590690 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:34.590819 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:34.591240 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:34.591243 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:34.591244 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:34.591670 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:34.591839 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:34.592282 (+ 443us) server_negotiation.cc:299] Negotiation successful
0925 04:48:34.592589 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":156,"threads_started":1}
I20260925 04:48:34.594559 21381 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93679fe0417c465787479054a42cb1ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93679fe0417c465787479054a42cb1ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35829 } }
I20260925 04:48:34.594935 21381 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93679fe0417c465787479054a42cb1ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:48:34.595037 21381 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93679fe0417c465787479054a42cb1ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93679fe0417c465787479054a42cb1ce, State: Initialized, Role: FOLLOWER
I20260925 04:48:34.595572 21381 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93679fe0417c465787479054a42cb1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93679fe0417c465787479054a42cb1ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35829 } }
I20260925 04:48:34.595763 21381 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93679fe0417c465787479054a42cb1ce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:48:34.595844 21381 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93679fe0417c465787479054a42cb1ce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:48:34.595955 21381 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93679fe0417c465787479054a42cb1ce [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:48:34.597030 393 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:48:34.597275 21381 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93679fe0417c465787479054a42cb1ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93679fe0417c465787479054a42cb1ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35829 } }
I20260925 04:48:34.597651 21381 leader_election.cc:302] T 00000000000000000000000000000000 P 93679fe0417c465787479054a42cb1ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93679fe0417c465787479054a42cb1ce; no voters:
I20260925 04:48:34.598047 21381 leader_election.cc:288] T 00000000000000000000000000000000 P 93679fe0417c465787479054a42cb1ce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:48:34.598165 21386 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93679fe0417c465787479054a42cb1ce [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:48:34.598470 21386 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93679fe0417c465787479054a42cb1ce [term 1 LEADER]: Becoming Leader. State: Replica: 93679fe0417c465787479054a42cb1ce, State: Running, Role: LEADER
I20260925 04:48:34.598815 21386 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93679fe0417c465787479054a42cb1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93679fe0417c465787479054a42cb1ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35829 } }
I20260925 04:48:34.599538 21381 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93679fe0417c465787479054a42cb1ce [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:48:34.600090 21386 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93679fe0417c465787479054a42cb1ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93679fe0417c465787479054a42cb1ce. Latest consensus state: current_term: 1 leader_uuid: "93679fe0417c465787479054a42cb1ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93679fe0417c465787479054a42cb1ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35829 } } }
I20260925 04:48:34.600272 21386 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93679fe0417c465787479054a42cb1ce [sys.catalog]: This master's current role is: LEADER
I20260925 04:48:34.600531 21387 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93679fe0417c465787479054a42cb1ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "93679fe0417c465787479054a42cb1ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93679fe0417c465787479054a42cb1ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35829 } } }
I20260925 04:48:34.600631 21387 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93679fe0417c465787479054a42cb1ce [sys.catalog]: This master's current role is: LEADER
I20260925 04:48:34.600731 21399 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:48:34.604076 21399 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:48:34.610184 21399 catalog_manager.cc:1347] Generated new cluster ID: 1f33244ec89b4fa0a4ab042681f71be1
I20260925 04:48:34.610280 21399 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:48:34.613327 21384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:34.601832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.103:47773 (local address 127.0.0.1:35829)
0925 04:48:34.601934 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:34.601939 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:34.602284 (+ 345us) server_negotiation.cc:407] Connection header received
0925 04:48:34.602573 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:34.602579 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:34.602629 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:34.602684 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:34.603852 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:34.604462 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:34.605480 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:34.605722 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:34.605836 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:34.608975 (+ 3139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:34.608979 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:34.608981 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:48:34.609008 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:34.611340 (+ 2332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:34.611941 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:34.611944 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:34.611946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:34.612031 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:34.612460 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:34.612463 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:34.612465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:34.612649 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:34.612785 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:34.613086 (+ 301us) server_negotiation.cc:299] Negotiation successful
0925 04:48:34.613176 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 04:48:34.620162 21399 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:48:34.621165 21399 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:48:34.627002 21399 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 93679fe0417c465787479054a42cb1ce: Generated new TSK 0
I20260925 04:48:34.627709 21399 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:48:34.634472 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 21313
W20260925 04:48:34.641508 21298 connection.cc:535] client connection to 127.0.0.1:35829 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35829): Connection reset by peer (error 104)
2026-09-25T04:48:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1426 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790311449873301-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:48:34 dist-test-slave-kghz krb5kdc[21412](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:48:34 dist-test-slave-kghz krb5kdc[21412](info): set up 2 sockets
Sep 25 04:48:34 dist-test-slave-kghz krb5kdc[21412](info): commencing operation
krb5kdc: starting...
W20260925 04:48:35.682650 393 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790311449873301-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790311449873301-393-0/krb5kdc/test-user.keytab.
Sep 25 04:48:35 dist-test-slave-kghz krb5kdc[21412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311715, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:48:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:48:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790311449873301-393-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790311449873301-393-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 04:48:35.810189 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790311449873301-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790311449873301-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790311449873301-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43397
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790311449873301-393-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43397
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:48:35.924775 21425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:48:35.925035 21425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:48:35.925077 21425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:48:35.929066 21425 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:48:35.929136 21425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:48:35.929157 21425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:48:35.929173 21425 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:48:35.929188 21425 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:48:35.934470 21425 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790311449873301-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790311449873301-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43397
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790311449873301-393-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790311449873301-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:48:35.935472 21425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:48:35.936506 21425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:48:35.945186 21433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:48:35.945189 21430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:48:35.945194 21431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:48:35.945231 21425 server_base.cc:1034] running on GCE node
I20260925 04:48:35.946017 21425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:48:35.947190 21425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:48:35.948401 21425 hybrid_clock.cc:648] HybridClock initialized: now 1790311715948387 us; error 27 us; skew 500 ppm
Sep 25 04:48:35 dist-test-slave-kghz krb5kdc[21412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311715, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:48:35.958036 21425 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 04:48:35.959180 21425 webserver.cc:460] Webserver started at http://127.0.0.1:43293/ using document root <none> and password file <none>
I20260925 04:48:35.959810 21425 fs_manager.cc:359] Metadata directory not provided
I20260925 04:48:35.959914 21425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:48:35.960155 21425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:48:35.962342 21425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790311449873301-393-0/minicluster-data/master-0/data/instance:
uuid: "ef2633dea3ed49a0bece7c087540c9d9"
format_stamp: "Formatted at 2026-09-25 04:48:35 on dist-test-slave-kghz"
server_key: "81ab68d2f7900c964fc6302981b793fa"
server_key_iv: "eaca3e7c2b918f54a852971cd7f73ae6"
server_key_version: "encryptionkey@0"
I20260925 04:48:35.962932 21425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790311449873301-393-0/minicluster-data/master-0/wal/instance:
uuid: "ef2633dea3ed49a0bece7c087540c9d9"
format_stamp: "Formatted at 2026-09-25 04:48:35 on dist-test-slave-kghz"
server_key: "81ab68d2f7900c964fc6302981b793fa"
server_key_iv: "eaca3e7c2b918f54a852971cd7f73ae6"
server_key_version: "encryptionkey@0"
I20260925 04:48:35.966830 21425 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260925 04:48:35.969291 21440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:48:35.970603 21425 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 04:48:35.970732 21425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790311449873301-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790311449873301-393-0/minicluster-data/master-0/wal
uuid: "ef2633dea3ed49a0bece7c087540c9d9"
format_stamp: "Formatted at 2026-09-25 04:48:35 on dist-test-slave-kghz"
server_key: "81ab68d2f7900c964fc6302981b793fa"
server_key_iv: "eaca3e7c2b918f54a852971cd7f73ae6"
server_key_version: "encryptionkey@0"
I20260925 04:48:35.970840 21425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790311449873301-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790311449873301-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790311449873301-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:48:35.986560 21425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:48:35.989781 21425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:48:35.990000 21425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:48:35.997157 21425 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43397
I20260925 04:48:35.997234 21492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43397 every 8 connection(s)
I20260925 04:48:35.998169 21425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
I20260925 04:48:36.000902 21493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:48:36.006335 21493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef2633dea3ed49a0bece7c087540c9d9: Bootstrap starting.
I20260925 04:48:36.006927 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 21425
I20260925 04:48:36.007066 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790311449873301-393-0/minicluster-data/master-0/wal/instance
I20260925 04:48:36.007330 393 external_mini_cluster.cc:1428] Setting key ab8142f8ddba26bc65ec1a03ab9db9d0
I20260925 04:48:36.009116 21493 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef2633dea3ed49a0bece7c087540c9d9: Neither blocks nor log segments found. Creating new log.
I20260925 04:48:36.009862 21493 log.cc:824] T 00000000000000000000000000000000 P ef2633dea3ed49a0bece7c087540c9d9: Log is configured to *not* fsync() on all Append() calls
I20260925 04:48:36.011752 21493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef2633dea3ed49a0bece7c087540c9d9: No bootstrap required, opened a new log
I20260925 04:48:36.016070 21493 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef2633dea3ed49a0bece7c087540c9d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef2633dea3ed49a0bece7c087540c9d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43397 } }
I20260925 04:48:36.016324 21493 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef2633dea3ed49a0bece7c087540c9d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:48:36.016404 21493 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef2633dea3ed49a0bece7c087540c9d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef2633dea3ed49a0bece7c087540c9d9, State: Initialized, Role: FOLLOWER
I20260925 04:48:36.016846 21496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:36.008651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.103:42149 (local address 127.0.0.1:43397)
0925 04:48:36.009077 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:36.009093 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:36.009130 (+ 37us) server_negotiation.cc:407] Connection header received
0925 04:48:36.009818 (+ 688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:36.009835 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:36.010298 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:36.010658 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:36.011069 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:36.012167 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:36.013392 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:36.013886 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:36.014005 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:36.014987 (+ 982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:36.014992 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:36.015006 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:48:36.015407 (+ 401us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:36.015469 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:48:36.015492 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:36.015590 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:36.015803 (+ 213us) server_negotiation.cc:299] Negotiation successful
0925 04:48:36.015968 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":113,"threads_started":1}
I20260925 04:48:36.016887 21493 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef2633dea3ed49a0bece7c087540c9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef2633dea3ed49a0bece7c087540c9d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43397 } }
I20260925 04:48:36.017105 21493 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef2633dea3ed49a0bece7c087540c9d9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:48:36.017179 21493 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef2633dea3ed49a0bece7c087540c9d9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:48:36.017277 21493 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef2633dea3ed49a0bece7c087540c9d9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:48:36.018384 21493 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef2633dea3ed49a0bece7c087540c9d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef2633dea3ed49a0bece7c087540c9d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43397 } }
I20260925 04:48:36.018760 21493 leader_election.cc:302] T 00000000000000000000000000000000 P ef2633dea3ed49a0bece7c087540c9d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef2633dea3ed49a0bece7c087540c9d9; no voters:
I20260925 04:48:36.019120 21493 leader_election.cc:288] T 00000000000000000000000000000000 P ef2633dea3ed49a0bece7c087540c9d9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:48:36.019196 21498 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef2633dea3ed49a0bece7c087540c9d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:48:36.019398 21498 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef2633dea3ed49a0bece7c087540c9d9 [term 1 LEADER]: Becoming Leader. State: Replica: ef2633dea3ed49a0bece7c087540c9d9, State: Running, Role: LEADER
I20260925 04:48:36.019737 21498 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef2633dea3ed49a0bece7c087540c9d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef2633dea3ed49a0bece7c087540c9d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43397 } }
I20260925 04:48:36.019972 393 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:48:36.020414 21493 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef2633dea3ed49a0bece7c087540c9d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:48:36.021472 21500 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef2633dea3ed49a0bece7c087540c9d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef2633dea3ed49a0bece7c087540c9d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef2633dea3ed49a0bece7c087540c9d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43397 } } }
I20260925 04:48:36.021601 21500 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef2633dea3ed49a0bece7c087540c9d9 [sys.catalog]: This master's current role is: LEADER
I20260925 04:48:36.021898 21499 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef2633dea3ed49a0bece7c087540c9d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef2633dea3ed49a0bece7c087540c9d9. Latest consensus state: current_term: 1 leader_uuid: "ef2633dea3ed49a0bece7c087540c9d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef2633dea3ed49a0bece7c087540c9d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43397 } } }
I20260925 04:48:36.022025 21499 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef2633dea3ed49a0bece7c087540c9d9 [sys.catalog]: This master's current role is: LEADER
I20260925 04:48:36.022006 21511 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:48:36.025379 21511 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:48:36.028168 21496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:36.022858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.103:37719 (local address 127.0.0.1:43397)
0925 04:48:36.022968 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:36.022973 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:36.023525 (+ 552us) server_negotiation.cc:407] Connection header received
0925 04:48:36.023712 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:36.023715 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:36.023765 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:36.023824 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:36.024259 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:36.025057 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:36.026329 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:36.026572 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:36.026689 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:36.027649 (+ 960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:36.027653 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:36.027656 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:48:36.027670 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:36.027698 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:48:36.027711 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:36.027821 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:36.027992 (+ 171us) server_negotiation.cc:299] Negotiation successful
0925 04:48:36.028028 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 04:48:36.031162 21511 catalog_manager.cc:1347] Generated new cluster ID: 156868ae59114fb384a0b07d65654977
I20260925 04:48:36.031224 21511 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:48:36.058418 21511 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:48:36.059454 21511 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:48:36.064960 21511 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef2633dea3ed49a0bece7c087540c9d9: Generated new TSK 0
I20260925 04:48:36.065665 21511 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:48:36.074815 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 21425
W20260925 04:48:36.080075 21410 connection.cc:535] client connection to 127.0.0.1:43397 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43397): Connection reset by peer (error 104)
2026-09-25T04:48:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1438 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790311449873301-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:48:36 dist-test-slave-kghz krb5kdc[21524](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:48:36 dist-test-slave-kghz krb5kdc[21524](info): set up 2 sockets
Sep 25 04:48:36 dist-test-slave-kghz krb5kdc[21524](info): commencing operation
krb5kdc: starting...
W20260925 04:48:37.111203 393 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790311449873301-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790311449873301-393-0/krb5kdc/test-user.keytab.
Sep 25 04:48:37 dist-test-slave-kghz krb5kdc[21524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:48:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:48:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790311449873301-393-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790311449873301-393-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 04:48:37.238171 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790311449873301-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790311449873301-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790311449873301-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33747
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790311449873301-393-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33747
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:48:37.349435 21537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:48:37.349753 21537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:48:37.349825 21537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:48:37.353984 21537 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:48:37.354074 21537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:48:37.354110 21537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:48:37.354166 21537 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:48:37.354206 21537 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:48:37.359632 21537 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790311449873301-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790311449873301-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33747
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790311449873301-393-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21537
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790311449873301-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:48:37.360764 21537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:48:37.361814 21537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:48:37.370746 21543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:48:37.370859 21542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:48:37.370746 21545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:48:37.371338 21537 server_base.cc:1034] running on GCE node
I20260925 04:48:37.372038 21537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:48:37.373188 21537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:48:37.374337 21537 hybrid_clock.cc:648] HybridClock initialized: now 1790311717374315 us; error 33 us; skew 500 ppm
Sep 25 04:48:37 dist-test-slave-kghz krb5kdc[21524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311717, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:48:37.377089 21537 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 04:48:37.378036 21537 webserver.cc:460] Webserver started at http://127.0.0.1:41119/ using document root <none> and password file <none>
I20260925 04:48:37.378654 21537 fs_manager.cc:359] Metadata directory not provided
I20260925 04:48:37.378746 21537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:48:37.378976 21537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:48:37.381004 21537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790311449873301-393-0/minicluster-data/master-0/data/instance:
uuid: "31ba495c839e4ea48343ccdd238837d3"
format_stamp: "Formatted at 2026-09-25 04:48:37 on dist-test-slave-kghz"
server_key: "fb188177d48836d17bf0deeb7da21f09"
server_key_iv: "3a09a903a7602741b73c0e107acf0922"
server_key_version: "encryptionkey@0"
I20260925 04:48:37.381561 21537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790311449873301-393-0/minicluster-data/master-0/wal/instance:
uuid: "31ba495c839e4ea48343ccdd238837d3"
format_stamp: "Formatted at 2026-09-25 04:48:37 on dist-test-slave-kghz"
server_key: "fb188177d48836d17bf0deeb7da21f09"
server_key_iv: "3a09a903a7602741b73c0e107acf0922"
server_key_version: "encryptionkey@0"
I20260925 04:48:37.385227 21537 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 04:48:37.387668 21552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:48:37.388895 21537 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 04:48:37.389037 21537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790311449873301-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790311449873301-393-0/minicluster-data/master-0/wal
uuid: "31ba495c839e4ea48343ccdd238837d3"
format_stamp: "Formatted at 2026-09-25 04:48:37 on dist-test-slave-kghz"
server_key: "fb188177d48836d17bf0deeb7da21f09"
server_key_iv: "3a09a903a7602741b73c0e107acf0922"
server_key_version: "encryptionkey@0"
I20260925 04:48:37.389154 21537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790311449873301-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790311449873301-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790311449873301-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:48:37.393860 21537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:48:37.397318 21537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:48:37.397539 21537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:48:37.405153 21537 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33747
I20260925 04:48:37.405179 21604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33747 every 8 connection(s)
I20260925 04:48:37.406170 21537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
I20260925 04:48:37.409013 21605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:48:37.414711 21605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31ba495c839e4ea48343ccdd238837d3: Bootstrap starting.
I20260925 04:48:37.414746 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 21537
I20260925 04:48:37.414943 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790311449873301-393-0/minicluster-data/master-0/wal/instance
I20260925 04:48:37.415237 393 external_mini_cluster.cc:1428] Setting key d132ab5dfea21cfb51daf4c157883523
I20260925 04:48:37.417515 21605 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31ba495c839e4ea48343ccdd238837d3: Neither blocks nor log segments found. Creating new log.
I20260925 04:48:37.418463 21605 log.cc:824] T 00000000000000000000000000000000 P 31ba495c839e4ea48343ccdd238837d3: Log is configured to *not* fsync() on all Append() calls
I20260925 04:48:37.420849 21605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31ba495c839e4ea48343ccdd238837d3: No bootstrap required, opened a new log
I20260925 04:48:37.421236 21608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:37.416657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.103:56087 (local address 127.0.0.1:33747)
0925 04:48:37.417107 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:37.417125 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:37.417167 (+ 42us) server_negotiation.cc:407] Connection header received
0925 04:48:37.418083 (+ 916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:37.418106 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:37.418516 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:37.418946 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:37.419829 (+ 883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:37.419834 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:37.419847 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:48:37.419858 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:37.419917 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:48:37.419936 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:37.420053 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:37.420213 (+ 160us) server_negotiation.cc:299] Negotiation successful
0925 04:48:37.420423 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":126,"threads_started":1}
I20260925 04:48:37.424288 393 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:48:37.425338 21605 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31ba495c839e4ea48343ccdd238837d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31ba495c839e4ea48343ccdd238837d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33747 } }
I20260925 04:48:37.425611 21605 raft_consensus.cc:374] T 00000000000000000000000000000000 P 31ba495c839e4ea48343ccdd238837d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:48:37.425683 21605 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31ba495c839e4ea48343ccdd238837d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31ba495c839e4ea48343ccdd238837d3, State: Initialized, Role: FOLLOWER
I20260925 04:48:37.426144 21605 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31ba495c839e4ea48343ccdd238837d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31ba495c839e4ea48343ccdd238837d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33747 } }
I20260925 04:48:37.426339 21605 raft_consensus.cc:388] T 00000000000000000000000000000000 P 31ba495c839e4ea48343ccdd238837d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:48:37.426412 21605 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31ba495c839e4ea48343ccdd238837d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:48:37.426564 21605 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31ba495c839e4ea48343ccdd238837d3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:48:37.427930 21605 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31ba495c839e4ea48343ccdd238837d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31ba495c839e4ea48343ccdd238837d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33747 } }
I20260925 04:48:37.428148 21608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:37.426407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.103:51155 (local address 127.0.0.1:33747)
0925 04:48:37.426546 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:37.426553 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:37.426728 (+ 175us) server_negotiation.cc:407] Connection header received
0925 04:48:37.426896 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:37.426898 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:37.426929 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:37.426995 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:37.427720 (+ 725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:37.427723 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:37.427725 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:48:37.427735 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:37.427759 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:48:37.427769 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:37.427837 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:37.427963 (+ 126us) server_negotiation.cc:299] Negotiation successful
0925 04:48:37.427995 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 04:48:37.428323 21605 leader_election.cc:302] T 00000000000000000000000000000000 P 31ba495c839e4ea48343ccdd238837d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31ba495c839e4ea48343ccdd238837d3; no voters:
I20260925 04:48:37.428673 21605 leader_election.cc:288] T 00000000000000000000000000000000 P 31ba495c839e4ea48343ccdd238837d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:48:37.428838 21615 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31ba495c839e4ea48343ccdd238837d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:48:37.429203 21615 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31ba495c839e4ea48343ccdd238837d3 [term 1 LEADER]: Becoming Leader. State: Replica: 31ba495c839e4ea48343ccdd238837d3, State: Running, Role: LEADER
I20260925 04:48:37.429530 21615 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31ba495c839e4ea48343ccdd238837d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31ba495c839e4ea48343ccdd238837d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33747 } }
I20260925 04:48:37.429942 21605 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31ba495c839e4ea48343ccdd238837d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:48:37.430675 21615 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31ba495c839e4ea48343ccdd238837d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31ba495c839e4ea48343ccdd238837d3. Latest consensus state: current_term: 1 leader_uuid: "31ba495c839e4ea48343ccdd238837d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31ba495c839e4ea48343ccdd238837d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33747 } } }
I20260925 04:48:37.430845 21615 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31ba495c839e4ea48343ccdd238837d3 [sys.catalog]: This master's current role is: LEADER
I20260925 04:48:37.431119 21616 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31ba495c839e4ea48343ccdd238837d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "31ba495c839e4ea48343ccdd238837d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31ba495c839e4ea48343ccdd238837d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33747 } } }
I20260925 04:48:37.431311 21616 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31ba495c839e4ea48343ccdd238837d3 [sys.catalog]: This master's current role is: LEADER
I20260925 04:48:37.431255 21622 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:48:37.434165 21622 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:48:37.439886 21622 catalog_manager.cc:1347] Generated new cluster ID: 23558fcf3b0748daab65f4c4172cb6ca
I20260925 04:48:37.439983 21622 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:48:37.460158 21622 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:48:37.460538 21622 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:48:37.469203 21622 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 31ba495c839e4ea48343ccdd238837d3: Generated new TSK 0
I20260925 04:48:37.470084 21622 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:48:37.473312 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 21537
2026-09-25T04:48:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1398 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790311449873301-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:48:37 dist-test-slave-kghz krb5kdc[21636](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:48:37 dist-test-slave-kghz krb5kdc[21636](info): set up 2 sockets
Sep 25 04:48:37 dist-test-slave-kghz krb5kdc[21636](info): commencing operation
krb5kdc: starting...
W20260925 04:48:38.502705 393 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790311449873301-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790311449873301-393-0/krb5kdc/test-user.keytab.
Sep 25 04:48:38 dist-test-slave-kghz krb5kdc[21636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:48:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:48:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790311449873301-393-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790311449873301-393-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 04:48:38.630107 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790311449873301-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790311449873301-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790311449873301-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36275
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790311449873301-393-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36275
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:48:38.740924 21649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:48:38.741199 21649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:48:38.741241 21649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:48:38.745327 21649 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:48:38.745447 21649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:48:38.745484 21649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:48:38.745522 21649 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:48:38.745553 21649 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:48:38.751034 21649 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790311449873301-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790311449873301-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36275
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790311449873301-393-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21649
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790311449873301-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:48:38.752147 21649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:48:38.753213 21649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:48:38.761868 21655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:48:38.761814 21654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:48:38.761977 21649 server_base.cc:1034] running on GCE node
W20260925 04:48:38.762243 21657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:48:38.762789 21649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:48:38.763880 21649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:48:38.765127 21649 hybrid_clock.cc:648] HybridClock initialized: now 1790311718765114 us; error 26 us; skew 500 ppm
Sep 25 04:48:38 dist-test-slave-kghz krb5kdc[21636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311718, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:48:38.767980 21649 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 04:48:38.768916 21649 webserver.cc:460] Webserver started at http://127.0.0.1:33359/ using document root <none> and password file <none>
I20260925 04:48:38.769480 21649 fs_manager.cc:359] Metadata directory not provided
I20260925 04:48:38.769572 21649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:48:38.769802 21649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:48:38.771924 21649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790311449873301-393-0/minicluster-data/master-0/data/instance:
uuid: "45c675ec2a1745a9b867773fcd9700d7"
format_stamp: "Formatted at 2026-09-25 04:48:38 on dist-test-slave-kghz"
server_key: "80d78a9b978fda80b1659c87a2959d7f"
server_key_iv: "07eae1001f01c7e9e177779cf9cbb311"
server_key_version: "encryptionkey@0"
I20260925 04:48:38.772511 21649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790311449873301-393-0/minicluster-data/master-0/wal/instance:
uuid: "45c675ec2a1745a9b867773fcd9700d7"
format_stamp: "Formatted at 2026-09-25 04:48:38 on dist-test-slave-kghz"
server_key: "80d78a9b978fda80b1659c87a2959d7f"
server_key_iv: "07eae1001f01c7e9e177779cf9cbb311"
server_key_version: "encryptionkey@0"
I20260925 04:48:38.776292 21649 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 04:48:38.778805 21664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:48:38.780085 21649 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 04:48:38.780229 21649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790311449873301-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790311449873301-393-0/minicluster-data/master-0/wal
uuid: "45c675ec2a1745a9b867773fcd9700d7"
format_stamp: "Formatted at 2026-09-25 04:48:38 on dist-test-slave-kghz"
server_key: "80d78a9b978fda80b1659c87a2959d7f"
server_key_iv: "07eae1001f01c7e9e177779cf9cbb311"
server_key_version: "encryptionkey@0"
I20260925 04:48:38.780377 21649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790311449873301-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790311449873301-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790311449873301-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:48:38.820833 21649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:48:38.824419 21649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:48:38.824644 21649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:48:38.831986 21649 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36275
I20260925 04:48:38.832019 21716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36275 every 8 connection(s)
I20260925 04:48:38.832964 21649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
I20260925 04:48:38.835873 21717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:48:38.837015 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 21649
I20260925 04:48:38.837160 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790311449873301-393-0/minicluster-data/master-0/wal/instance
I20260925 04:48:38.837491 393 external_mini_cluster.cc:1428] Setting key aafda0b1bda5f0aa9b4fb6ad88bfb755
I20260925 04:48:38.842163 21717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45c675ec2a1745a9b867773fcd9700d7: Bootstrap starting.
I20260925 04:48:38.844429 21717 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45c675ec2a1745a9b867773fcd9700d7: Neither blocks nor log segments found. Creating new log.
I20260925 04:48:38.845178 21717 log.cc:824] T 00000000000000000000000000000000 P 45c675ec2a1745a9b867773fcd9700d7: Log is configured to *not* fsync() on all Append() calls
Sep 25 04:48:38 dist-test-slave-kghz krb5kdc[21636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 04:48:38.847349 21717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45c675ec2a1745a9b867773fcd9700d7: No bootstrap required, opened a new log
I20260925 04:48:38.852172 21717 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45c675ec2a1745a9b867773fcd9700d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45c675ec2a1745a9b867773fcd9700d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36275 } }
I20260925 04:48:38.852450 21717 raft_consensus.cc:374] T 00000000000000000000000000000000 P 45c675ec2a1745a9b867773fcd9700d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:48:38.852473 21720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:38.839096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.103:43741 (local address 127.0.0.1:36275)
0925 04:48:38.839549 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:38.839564 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:38.839601 (+ 37us) server_negotiation.cc:407] Connection header received
0925 04:48:38.840235 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:38.840251 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:38.840604 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:38.840939 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:38.841762 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:38.842834 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:38.843707 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:38.844184 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:38.844352 (+ 168us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:38.847047 (+ 2695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:38.847054 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:38.847070 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:48:38.847099 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:38.849161 (+ 2062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:38.849706 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:38.849713 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:38.849722 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:38.849889 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:38.850280 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:38.850283 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:38.850285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:38.850782 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:38.851029 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:38.851269 (+ 240us) server_negotiation.cc:299] Negotiation successful
0925 04:48:38.851505 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":126,"threads_started":1}
I20260925 04:48:38.852509 21717 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45c675ec2a1745a9b867773fcd9700d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45c675ec2a1745a9b867773fcd9700d7, State: Initialized, Role: FOLLOWER
I20260925 04:48:38.852957 21717 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45c675ec2a1745a9b867773fcd9700d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45c675ec2a1745a9b867773fcd9700d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36275 } }
I20260925 04:48:38.853118 21717 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45c675ec2a1745a9b867773fcd9700d7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:48:38.853184 21717 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45c675ec2a1745a9b867773fcd9700d7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:48:38.853288 21717 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45c675ec2a1745a9b867773fcd9700d7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:48:38.854513 21717 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45c675ec2a1745a9b867773fcd9700d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45c675ec2a1745a9b867773fcd9700d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36275 } }
I20260925 04:48:38.854889 21717 leader_election.cc:302] T 00000000000000000000000000000000 P 45c675ec2a1745a9b867773fcd9700d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45c675ec2a1745a9b867773fcd9700d7; no voters:
I20260925 04:48:38.855240 21717 leader_election.cc:288] T 00000000000000000000000000000000 P 45c675ec2a1745a9b867773fcd9700d7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:48:38.855348 21722 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45c675ec2a1745a9b867773fcd9700d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:48:38.855654 21722 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45c675ec2a1745a9b867773fcd9700d7 [term 1 LEADER]: Becoming Leader. State: Replica: 45c675ec2a1745a9b867773fcd9700d7, State: Running, Role: LEADER
I20260925 04:48:38.855847 393 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:48:38.856091 21722 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45c675ec2a1745a9b867773fcd9700d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45c675ec2a1745a9b867773fcd9700d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36275 } }
I20260925 04:48:38.856655 21717 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45c675ec2a1745a9b867773fcd9700d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:48:38.857821 21723 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45c675ec2a1745a9b867773fcd9700d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "45c675ec2a1745a9b867773fcd9700d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45c675ec2a1745a9b867773fcd9700d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36275 } } }
I20260925 04:48:38.857959 21723 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45c675ec2a1745a9b867773fcd9700d7 [sys.catalog]: This master's current role is: LEADER
I20260925 04:48:38.858338 21724 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45c675ec2a1745a9b867773fcd9700d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45c675ec2a1745a9b867773fcd9700d7. Latest consensus state: current_term: 1 leader_uuid: "45c675ec2a1745a9b867773fcd9700d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45c675ec2a1745a9b867773fcd9700d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36275 } } }
I20260925 04:48:38.858397 21735 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:48:38.858466 21724 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45c675ec2a1745a9b867773fcd9700d7 [sys.catalog]: This master's current role is: LEADER
I20260925 04:48:38.861632 21735 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:48:38.867493 21735 catalog_manager.cc:1347] Generated new cluster ID: bb81d723b45a4f95895b5d4aafdd5e18
I20260925 04:48:38.868001 21735 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:48:38.868515 21720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:38.858306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.103:44527 (local address 127.0.0.1:36275)
0925 04:48:38.858433 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:38.858438 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:38.859354 (+ 916us) server_negotiation.cc:407] Connection header received
0925 04:48:38.859506 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:38.859510 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:38.859559 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:38.859648 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:38.860490 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:38.861232 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:38.862384 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:38.862705 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:38.862810 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:38.864516 (+ 1706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:38.864520 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:38.864522 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:48:38.864546 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:38.866431 (+ 1885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:38.866976 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:38.866980 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:38.866982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:38.867070 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:38.867473 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:38.867476 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:38.867478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:38.867744 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:38.867901 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:38.868257 (+ 356us) server_negotiation.cc:299] Negotiation successful
0925 04:48:38.868352 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 04:48:38.882283 21735 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:48:38.883255 21735 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:48:38.891242 21735 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 45c675ec2a1745a9b867773fcd9700d7: Generated new TSK 0
I20260925 04:48:38.891970 21735 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:48:38.896134 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 21649
W20260925 04:48:38.901535 21634 connection.cc:535] client connection to 127.0.0.1:36275 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36275): Connection reset by peer (error 104)
2026-09-25T04:48:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1423 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790311449873301-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:48:38 dist-test-slave-kghz krb5kdc[21748](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:48:38 dist-test-slave-kghz krb5kdc[21748](info): set up 2 sockets
Sep 25 04:48:38 dist-test-slave-kghz krb5kdc[21748](info): commencing operation
krb5kdc: starting...
W20260925 04:48:39.938560 393 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790311449873301-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790311449873301-393-0/krb5kdc/test-user.keytab.
Sep 25 04:48:40 dist-test-slave-kghz krb5kdc[21748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311720, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:48:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:48:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790311449873301-393-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790311449873301-393-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 04:48:40.066427 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790311449873301-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790311449873301-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790311449873301-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39761
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790311449873301-393-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39761
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:48:40.178416 21761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:48:40.178699 21761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:48:40.178742 21761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:48:40.182690 21761 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:48:40.182744 21761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:48:40.182762 21761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:48:40.182780 21761 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:48:40.182794 21761 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:48:40.187930 21761 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790311449873301-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790311449873301-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39761
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790311449873301-393-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21761
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790311449873301-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:48:40.188932 21761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:48:40.189937 21761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:48:40.198882 21766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:48:40.198889 21767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:48:40.198941 21761 server_base.cc:1034] running on GCE node
W20260925 04:48:40.198879 21769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:48:40.199697 21761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:48:40.200845 21761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:48:40.202050 21761 hybrid_clock.cc:648] HybridClock initialized: now 1790311720202036 us; error 25 us; skew 500 ppm
Sep 25 04:48:40 dist-test-slave-kghz krb5kdc[21748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311720, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:48:40.204936 21761 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 04:48:40.205894 21761 webserver.cc:460] Webserver started at http://127.0.0.1:44103/ using document root <none> and password file <none>
I20260925 04:48:40.206562 21761 fs_manager.cc:359] Metadata directory not provided
I20260925 04:48:40.206657 21761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:48:40.206902 21761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:48:40.208987 21761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790311449873301-393-0/minicluster-data/master-0/data/instance:
uuid: "f305493f5114461dba1a0961868ae18a"
format_stamp: "Formatted at 2026-09-25 04:48:40 on dist-test-slave-kghz"
server_key: "bb1c39582f0244ed4999197a325fa1e8"
server_key_iv: "475fa0424482e4d37a6c756c66c2ed68"
server_key_version: "encryptionkey@0"
I20260925 04:48:40.209556 21761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790311449873301-393-0/minicluster-data/master-0/wal/instance:
uuid: "f305493f5114461dba1a0961868ae18a"
format_stamp: "Formatted at 2026-09-25 04:48:40 on dist-test-slave-kghz"
server_key: "bb1c39582f0244ed4999197a325fa1e8"
server_key_iv: "475fa0424482e4d37a6c756c66c2ed68"
server_key_version: "encryptionkey@0"
I20260925 04:48:40.213199 21761 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 04:48:40.215675 21776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:48:40.216915 21761 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 04:48:40.217056 21761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790311449873301-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790311449873301-393-0/minicluster-data/master-0/wal
uuid: "f305493f5114461dba1a0961868ae18a"
format_stamp: "Formatted at 2026-09-25 04:48:40 on dist-test-slave-kghz"
server_key: "bb1c39582f0244ed4999197a325fa1e8"
server_key_iv: "475fa0424482e4d37a6c756c66c2ed68"
server_key_version: "encryptionkey@0"
I20260925 04:48:40.217168 21761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790311449873301-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790311449873301-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790311449873301-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:48:40.253446 21761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:48:40.256608 21761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:48:40.256847 21761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:48:40.264309 21761 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39761
I20260925 04:48:40.264320 21828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39761 every 8 connection(s)
I20260925 04:48:40.265280 21761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
I20260925 04:48:40.268076 21829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:48:40.273015 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 21761
I20260925 04:48:40.273165 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790311449873301-393-0/minicluster-data/master-0/wal/instance
I20260925 04:48:40.273451 393 external_mini_cluster.cc:1428] Setting key 9136137205286ec763b3335018758bc2
I20260925 04:48:40.273989 21829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f305493f5114461dba1a0961868ae18a: Bootstrap starting.
I20260925 04:48:40.276427 21829 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f305493f5114461dba1a0961868ae18a: Neither blocks nor log segments found. Creating new log.
I20260925 04:48:40.277159 21829 log.cc:824] T 00000000000000000000000000000000 P f305493f5114461dba1a0961868ae18a: Log is configured to *not* fsync() on all Append() calls
I20260925 04:48:40.279106 21829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f305493f5114461dba1a0961868ae18a: No bootstrap required, opened a new log
I20260925 04:48:40.282305 21832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:40.274875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.103:57023 (local address 127.0.0.1:39761)
0925 04:48:40.275290 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:40.275306 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:40.275344 (+ 38us) server_negotiation.cc:407] Connection header received
0925 04:48:40.275969 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:40.275986 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:40.276324 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:40.276661 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:40.277091 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:40.278121 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:40.279048 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:40.279570 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:40.279750 (+ 180us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:40.280643 (+ 893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:40.280648 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:40.280660 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:48:40.281038 (+ 378us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:40.281089 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:48:40.281111 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:40.281253 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:40.281440 (+ 187us) server_negotiation.cc:299] Negotiation successful
0925 04:48:40.281574 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":142,"threads_started":1}
I20260925 04:48:40.282909 21829 raft_consensus.cc:348] T 00000000000000000000000000000000 P f305493f5114461dba1a0961868ae18a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f305493f5114461dba1a0961868ae18a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39761 } }
I20260925 04:48:40.283120 21829 raft_consensus.cc:374] T 00000000000000000000000000000000 P f305493f5114461dba1a0961868ae18a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:48:40.283179 21829 raft_consensus.cc:729] T 00000000000000000000000000000000 P f305493f5114461dba1a0961868ae18a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f305493f5114461dba1a0961868ae18a, State: Initialized, Role: FOLLOWER
I20260925 04:48:40.283663 21829 consensus_queue.cc:260] T 00000000000000000000000000000000 P f305493f5114461dba1a0961868ae18a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f305493f5114461dba1a0961868ae18a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39761 } }
I20260925 04:48:40.283823 21829 raft_consensus.cc:388] T 00000000000000000000000000000000 P f305493f5114461dba1a0961868ae18a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:48:40.283906 21829 raft_consensus.cc:482] T 00000000000000000000000000000000 P f305493f5114461dba1a0961868ae18a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:48:40.284032 21829 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f305493f5114461dba1a0961868ae18a [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:48:40.285027 21829 raft_consensus.cc:504] T 00000000000000000000000000000000 P f305493f5114461dba1a0961868ae18a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f305493f5114461dba1a0961868ae18a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39761 } }
I20260925 04:48:40.285274 393 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:48:40.285415 21829 leader_election.cc:302] T 00000000000000000000000000000000 P f305493f5114461dba1a0961868ae18a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f305493f5114461dba1a0961868ae18a; no voters:
I20260925 04:48:40.285722 21829 leader_election.cc:288] T 00000000000000000000000000000000 P f305493f5114461dba1a0961868ae18a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:48:40.285818 21834 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f305493f5114461dba1a0961868ae18a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:48:40.286128 21834 raft_consensus.cc:686] T 00000000000000000000000000000000 P f305493f5114461dba1a0961868ae18a [term 1 LEADER]: Becoming Leader. State: Replica: f305493f5114461dba1a0961868ae18a, State: Running, Role: LEADER
I20260925 04:48:40.286487 21834 consensus_queue.cc:237] T 00000000000000000000000000000000 P f305493f5114461dba1a0961868ae18a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f305493f5114461dba1a0961868ae18a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39761 } }
I20260925 04:48:40.287112 21829 sys_catalog.cc:564] T 00000000000000000000000000000000 P f305493f5114461dba1a0961868ae18a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:48:40.287865 21835 sys_catalog.cc:455] T 00000000000000000000000000000000 P f305493f5114461dba1a0961868ae18a [sys.catalog]: SysCatalogTable state changed. Reason: New leader f305493f5114461dba1a0961868ae18a. Latest consensus state: current_term: 1 leader_uuid: "f305493f5114461dba1a0961868ae18a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f305493f5114461dba1a0961868ae18a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39761 } } }
I20260925 04:48:40.288053 21835 sys_catalog.cc:458] T 00000000000000000000000000000000 P f305493f5114461dba1a0961868ae18a [sys.catalog]: This master's current role is: LEADER
I20260925 04:48:40.288237 21836 sys_catalog.cc:455] T 00000000000000000000000000000000 P f305493f5114461dba1a0961868ae18a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f305493f5114461dba1a0961868ae18a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f305493f5114461dba1a0961868ae18a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39761 } } }
I20260925 04:48:40.288352 21836 sys_catalog.cc:458] T 00000000000000000000000000000000 P f305493f5114461dba1a0961868ae18a [sys.catalog]: This master's current role is: LEADER
W20260925 04:48:40.292299 21854 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f305493f5114461dba1a0961868ae18a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 04:48:40.292418 21854 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 04:48:40.292505 21848 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:48:40.292888 21832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:40.287594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.103:36723 (local address 127.0.0.1:39761)
0925 04:48:40.287693 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:40.287696 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:40.288744 (+ 1048us) server_negotiation.cc:407] Connection header received
0925 04:48:40.288918 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:40.288922 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:40.288967 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:40.289020 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:40.289463 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:40.290201 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:40.291103 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:40.291342 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:40.291461 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:40.292392 (+ 931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:40.292395 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:40.292398 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:48:40.292422 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:40.292451 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:48:40.292466 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:40.292554 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:40.292714 (+ 160us) server_negotiation.cc:299] Negotiation successful
0925 04:48:40.292749 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 04:48:40.293426 21848 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:48:40.299381 21848 catalog_manager.cc:1347] Generated new cluster ID: 1696d10c628440e49ad50f0dcd37fafa
I20260925 04:48:40.299463 21848 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:48:40.314831 21848 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:48:40.315783 21848 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:48:40.325595 21848 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f305493f5114461dba1a0961868ae18a: Generated new TSK 0
I20260925 04:48:40.326352 21848 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:48:40.332311 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 21761
W20260925 04:48:40.337074 21746 connection.cc:535] client connection to 127.0.0.1:39761 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39761): Connection reset by peer (error 104)
2026-09-25T04:48:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1435 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790311449873301-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:48:40 dist-test-slave-kghz krb5kdc[21860](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:48:40 dist-test-slave-kghz krb5kdc[21860](info): set up 2 sockets
Sep 25 04:48:40 dist-test-slave-kghz krb5kdc[21860](info): commencing operation
krb5kdc: starting...
W20260925 04:48:41.370618 393 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790311449873301-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790311449873301-393-0/krb5kdc/test-user.keytab.
Sep 25 04:48:41 dist-test-slave-kghz krb5kdc[21860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311721, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:48:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:48:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790311449873301-393-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790311449873301-393-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 04:48:41.497596 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790311449873301-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790311449873301-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790311449873301-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46663
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790311449873301-393-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46663
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:48:41.611967 21873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:48:41.612287 21873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:48:41.612365 21873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:48:41.616451 21873 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:48:41.616568 21873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:48:41.616607 21873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:48:41.616665 21873 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:48:41.616703 21873 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:48:41.622263 21873 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790311449873301-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790311449873301-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46663
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790311449873301-393-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21873
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790311449873301-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:48:41.623462 21873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:48:41.624531 21873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:48:41.634238 21878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:48:41.634431 21873 server_base.cc:1034] running on GCE node
W20260925 04:48:41.634289 21879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:48:41.634603 21881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:48:41.635205 21873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:48:41.636374 21873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:48:41.637559 21873 hybrid_clock.cc:648] HybridClock initialized: now 1790311721637525 us; error 48 us; skew 500 ppm
Sep 25 04:48:41 dist-test-slave-kghz krb5kdc[21860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311721, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:48:41.645522 21873 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 04:48:41.646658 21873 webserver.cc:460] Webserver started at http://127.0.0.1:42193/ using document root <none> and password file <none>
I20260925 04:48:41.647275 21873 fs_manager.cc:359] Metadata directory not provided
I20260925 04:48:41.647377 21873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:48:41.647635 21873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:48:41.649842 21873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790311449873301-393-0/minicluster-data/master-0/data/instance:
uuid: "d2d5eed6045549d4aa1a5e165555f607"
format_stamp: "Formatted at 2026-09-25 04:48:41 on dist-test-slave-kghz"
server_key: "6b2242ecc424342c590a835b2b37c4af"
server_key_iv: "92004ff8c1d582ad203104c75dbe86d1"
server_key_version: "encryptionkey@0"
I20260925 04:48:41.650502 21873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790311449873301-393-0/minicluster-data/master-0/wal/instance:
uuid: "d2d5eed6045549d4aa1a5e165555f607"
format_stamp: "Formatted at 2026-09-25 04:48:41 on dist-test-slave-kghz"
server_key: "6b2242ecc424342c590a835b2b37c4af"
server_key_iv: "92004ff8c1d582ad203104c75dbe86d1"
server_key_version: "encryptionkey@0"
I20260925 04:48:41.654691 21873 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 04:48:41.657437 21888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:48:41.658962 21873 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:48:41.659117 21873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790311449873301-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790311449873301-393-0/minicluster-data/master-0/wal
uuid: "d2d5eed6045549d4aa1a5e165555f607"
format_stamp: "Formatted at 2026-09-25 04:48:41 on dist-test-slave-kghz"
server_key: "6b2242ecc424342c590a835b2b37c4af"
server_key_iv: "92004ff8c1d582ad203104c75dbe86d1"
server_key_version: "encryptionkey@0"
I20260925 04:48:41.659253 21873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790311449873301-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790311449873301-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790311449873301-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:48:41.664327 21873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:48:41.667985 21873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:48:41.668201 21873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:48:41.675257 21873 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46663
I20260925 04:48:41.675350 21940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46663 every 8 connection(s)
I20260925 04:48:41.676250 21873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
I20260925 04:48:41.679019 21941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:48:41.684158 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 21873
I20260925 04:48:41.684304 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790311449873301-393-0/minicluster-data/master-0/wal/instance
I20260925 04:48:41.684605 393 external_mini_cluster.cc:1428] Setting key 410868c6ee0e1e067320a971011dee85
I20260925 04:48:41.684784 21941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2d5eed6045549d4aa1a5e165555f607: Bootstrap starting.
I20260925 04:48:41.687289 21941 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2d5eed6045549d4aa1a5e165555f607: Neither blocks nor log segments found. Creating new log.
I20260925 04:48:41.688148 21941 log.cc:824] T 00000000000000000000000000000000 P d2d5eed6045549d4aa1a5e165555f607: Log is configured to *not* fsync() on all Append() calls
I20260925 04:48:41.689916 21944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:41.686030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.103:39587 (local address 127.0.0.1:46663)
0925 04:48:41.686518 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:41.686533 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:41.686568 (+ 35us) server_negotiation.cc:407] Connection header received
0925 04:48:41.687202 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:41.687217 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:41.687569 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:41.687858 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:41.688754 (+ 896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:41.688760 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:41.688781 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:48:41.688792 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:41.688844 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:48:41.688863 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:41.688983 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:41.689082 (+ 99us) server_negotiation.cc:299] Negotiation successful
0925 04:48:41.689237 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":177,"threads_started":1}
I20260925 04:48:41.690407 21941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2d5eed6045549d4aa1a5e165555f607: No bootstrap required, opened a new log
I20260925 04:48:41.692720 393 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:48:41.694239 21941 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2d5eed6045549d4aa1a5e165555f607 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2d5eed6045549d4aa1a5e165555f607" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46663 } }
I20260925 04:48:41.694550 21941 raft_consensus.cc:374] T 00000000000000000000000000000000 P d2d5eed6045549d4aa1a5e165555f607 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:48:41.694629 21941 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2d5eed6045549d4aa1a5e165555f607 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d5eed6045549d4aa1a5e165555f607, State: Initialized, Role: FOLLOWER
I20260925 04:48:41.695142 21941 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2d5eed6045549d4aa1a5e165555f607 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2d5eed6045549d4aa1a5e165555f607" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46663 } }
I20260925 04:48:41.695317 21941 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2d5eed6045549d4aa1a5e165555f607 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:48:41.695416 21941 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2d5eed6045549d4aa1a5e165555f607 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:48:41.695533 21941 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2d5eed6045549d4aa1a5e165555f607 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:48:41.696568 21944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:41.694745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.103:52081 (local address 127.0.0.1:46663)
0925 04:48:41.694880 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:41.694884 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:41.695160 (+ 276us) server_negotiation.cc:407] Connection header received
0925 04:48:41.695252 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:41.695255 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:41.695285 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:41.695405 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:41.696135 (+ 730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:41.696138 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:41.696140 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:48:41.696149 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:41.696173 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:48:41.696183 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:41.696252 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:41.696368 (+ 116us) server_negotiation.cc:299] Negotiation successful
0925 04:48:41.696400 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 04:48:41.696856 21941 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2d5eed6045549d4aa1a5e165555f607 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2d5eed6045549d4aa1a5e165555f607" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46663 } }
I20260925 04:48:41.697201 21941 leader_election.cc:302] T 00000000000000000000000000000000 P d2d5eed6045549d4aa1a5e165555f607 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2d5eed6045549d4aa1a5e165555f607; no voters:
I20260925 04:48:41.697543 21941 leader_election.cc:288] T 00000000000000000000000000000000 P d2d5eed6045549d4aa1a5e165555f607 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:48:41.697641 21951 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2d5eed6045549d4aa1a5e165555f607 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:48:41.697911 21951 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2d5eed6045549d4aa1a5e165555f607 [term 1 LEADER]: Becoming Leader. State: Replica: d2d5eed6045549d4aa1a5e165555f607, State: Running, Role: LEADER
I20260925 04:48:41.698300 21951 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2d5eed6045549d4aa1a5e165555f607 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2d5eed6045549d4aa1a5e165555f607" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46663 } }
I20260925 04:48:41.699119 21941 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2d5eed6045549d4aa1a5e165555f607 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:48:41.699456 21953 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2d5eed6045549d4aa1a5e165555f607 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2d5eed6045549d4aa1a5e165555f607. Latest consensus state: current_term: 1 leader_uuid: "d2d5eed6045549d4aa1a5e165555f607" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2d5eed6045549d4aa1a5e165555f607" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46663 } } }
I20260925 04:48:41.699646 21953 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2d5eed6045549d4aa1a5e165555f607 [sys.catalog]: This master's current role is: LEADER
I20260925 04:48:41.699932 21952 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2d5eed6045549d4aa1a5e165555f607 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d2d5eed6045549d4aa1a5e165555f607" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2d5eed6045549d4aa1a5e165555f607" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46663 } } }
I20260925 04:48:41.700081 21952 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2d5eed6045549d4aa1a5e165555f607 [sys.catalog]: This master's current role is: LEADER
I20260925 04:48:41.701365 21955 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:48:41.704705 21955 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:48:41.711009 21955 catalog_manager.cc:1347] Generated new cluster ID: 293cf71ba2274f30bfa7222a06570958
I20260925 04:48:41.711094 21955 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:48:41.730134 21955 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:48:41.730512 21955 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:48:41.744102 21955 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d2d5eed6045549d4aa1a5e165555f607: Generated new TSK 0
I20260925 04:48:41.744993 21955 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:48:41.757704 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 21873
2026-09-25T04:48:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1426 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790311449873301-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:48:41 dist-test-slave-kghz krb5kdc[21972](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:48:41 dist-test-slave-kghz krb5kdc[21972](info): set up 2 sockets
Sep 25 04:48:41 dist-test-slave-kghz krb5kdc[21972](info): commencing operation
krb5kdc: starting...
W20260925 04:48:42.830624 393 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790311449873301-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790311449873301-393-0/krb5kdc/test-user.keytab.
Sep 25 04:48:42 dist-test-slave-kghz krb5kdc[21972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:48:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:48:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260925 04:48:42.962980 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790311449873301-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790311449873301-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790311449873301-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:39729
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:44525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:39729
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:48:43.082195 21985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:48:43.082546 21985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:48:43.082597 21985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:48:43.086751 21985 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:48:43.086820 21985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:48:43.086840 21985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:48:43.086858 21985 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:48:43.086872 21985 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:48:43.092224 21985 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:44525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790311449873301-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790311449873301-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:39729
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:39729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21985
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790311449873301-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:48:43.093255 21985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:48:43.094231 21985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:48:43.102918 21990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:48:43.102945 21985 server_base.cc:1034] running on GCE node
W20260925 04:48:43.103004 21991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:48:43.102950 21993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:48:43.103965 21985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:48:43.105094 21985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:48:43.106323 21985 hybrid_clock.cc:648] HybridClock initialized: now 1790311723106304 us; error 30 us; skew 500 ppm
Sep 25 04:48:43 dist-test-slave-kghz krb5kdc[21972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311723, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:48:43.109311 21985 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260925 04:48:43.110402 21985 webserver.cc:460] Webserver started at http://127.0.98.126:40785/ using document root <none> and password file <none>
I20260925 04:48:43.110998 21985 fs_manager.cc:359] Metadata directory not provided
I20260925 04:48:43.111099 21985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:48:43.111337 21985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:48:43.113500 21985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790311449873301-393-0/minicluster-data/master-0/data/instance:
uuid: "45f90b19124d4a039140d7bdc7cb7079"
format_stamp: "Formatted at 2026-09-25 04:48:43 on dist-test-slave-kghz"
server_key: "dbe16792810f20ac4c64e6f750358179"
server_key_iv: "c20b0775600d8aaa204cdc7c849a8e3c"
server_key_version: "encryptionkey@0"
I20260925 04:48:43.114112 21985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790311449873301-393-0/minicluster-data/master-0/wal/instance:
uuid: "45f90b19124d4a039140d7bdc7cb7079"
format_stamp: "Formatted at 2026-09-25 04:48:43 on dist-test-slave-kghz"
server_key: "dbe16792810f20ac4c64e6f750358179"
server_key_iv: "c20b0775600d8aaa204cdc7c849a8e3c"
server_key_version: "encryptionkey@0"
I20260925 04:48:43.118024 21985 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 04:48:43.120643 22000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:48:43.122068 21985 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 04:48:43.122232 21985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790311449873301-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790311449873301-393-0/minicluster-data/master-0/wal
uuid: "45f90b19124d4a039140d7bdc7cb7079"
format_stamp: "Formatted at 2026-09-25 04:48:43 on dist-test-slave-kghz"
server_key: "dbe16792810f20ac4c64e6f750358179"
server_key_iv: "c20b0775600d8aaa204cdc7c849a8e3c"
server_key_version: "encryptionkey@0"
I20260925 04:48:43.122416 21985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790311449873301-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790311449873301-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790311449873301-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:48:43.140385 21985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:48:43.144263 21985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:48:43.144498 21985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:48:43.151974 21985 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:39729
I20260925 04:48:43.152052 22052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:39729 every 8 connection(s)
I20260925 04:48:43.153020 21985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
I20260925 04:48:43.156478 22053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:48:43.159893 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 21985
I20260925 04:48:43.160063 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790311449873301-393-0/minicluster-data/master-0/wal/instance
I20260925 04:48:43.160333 393 external_mini_cluster.cc:1428] Setting key f1cb4db8ab250a86664eccdd7a1fab53
I20260925 04:48:43.162544 22053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45f90b19124d4a039140d7bdc7cb7079: Bootstrap starting.
I20260925 04:48:43.164726 22053 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45f90b19124d4a039140d7bdc7cb7079: Neither blocks nor log segments found. Creating new log.
I20260925 04:48:43.165490 22053 log.cc:824] T 00000000000000000000000000000000 P 45f90b19124d4a039140d7bdc7cb7079: Log is configured to *not* fsync() on all Append() calls
I20260925 04:48:43.167644 22053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45f90b19124d4a039140d7bdc7cb7079: No bootstrap required, opened a new log
Sep 25 04:48:43 dist-test-slave-kghz krb5kdc[21972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:48:43.172407 22053 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45f90b19124d4a039140d7bdc7cb7079 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f90b19124d4a039140d7bdc7cb7079" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39729 } }
I20260925 04:48:43.172716 22053 raft_consensus.cc:374] T 00000000000000000000000000000000 P 45f90b19124d4a039140d7bdc7cb7079 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:48:43.172801 22053 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45f90b19124d4a039140d7bdc7cb7079 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45f90b19124d4a039140d7bdc7cb7079, State: Initialized, Role: FOLLOWER
I20260925 04:48:43.173269 22053 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45f90b19124d4a039140d7bdc7cb7079 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f90b19124d4a039140d7bdc7cb7079" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39729 } }
I20260925 04:48:43.173465 22053 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45f90b19124d4a039140d7bdc7cb7079 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:48:43.173537 22053 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45f90b19124d4a039140d7bdc7cb7079 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:48:43.173640 22053 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45f90b19124d4a039140d7bdc7cb7079 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:48:43.174990 22053 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45f90b19124d4a039140d7bdc7cb7079 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f90b19124d4a039140d7bdc7cb7079" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39729 } }
I20260925 04:48:43.176578 22056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:43.161743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53283 (local address 127.0.98.126:39729)
0925 04:48:43.162191 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:43.162206 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:43.162320 (+ 114us) server_negotiation.cc:407] Connection header received
0925 04:48:43.163006 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:43.163024 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:43.163374 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:43.163713 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:43.164579 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:43.165669 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:43.166794 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:43.167306 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:43.167494 (+ 188us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:43.170406 (+ 2912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:43.170414 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:43.170439 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:48:43.170480 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:43.172676 (+ 2196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:43.173395 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:43.173401 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:43.173408 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:43.173566 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:43.173974 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:43.173977 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:43.173979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:43.174541 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:43.174761 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:43.175034 (+ 273us) server_negotiation.cc:299] Negotiation successful
0925 04:48:43.175399 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":123,"threads_started":1}
I20260925 04:48:43.176662 22053 leader_election.cc:302] T 00000000000000000000000000000000 P 45f90b19124d4a039140d7bdc7cb7079 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45f90b19124d4a039140d7bdc7cb7079; no voters:
I20260925 04:48:43.177011 22053 leader_election.cc:288] T 00000000000000000000000000000000 P 45f90b19124d4a039140d7bdc7cb7079 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:48:43.177134 22058 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45f90b19124d4a039140d7bdc7cb7079 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:48:43.177426 22058 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45f90b19124d4a039140d7bdc7cb7079 [term 1 LEADER]: Becoming Leader. State: Replica: 45f90b19124d4a039140d7bdc7cb7079, State: Running, Role: LEADER
I20260925 04:48:43.177837 22058 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45f90b19124d4a039140d7bdc7cb7079 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f90b19124d4a039140d7bdc7cb7079" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39729 } }
I20260925 04:48:43.178418 22053 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45f90b19124d4a039140d7bdc7cb7079 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:48:43.180320 22060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45f90b19124d4a039140d7bdc7cb7079 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45f90b19124d4a039140d7bdc7cb7079. Latest consensus state: current_term: 1 leader_uuid: "45f90b19124d4a039140d7bdc7cb7079" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f90b19124d4a039140d7bdc7cb7079" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39729 } } }
I20260925 04:48:43.180462 22060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45f90b19124d4a039140d7bdc7cb7079 [sys.catalog]: This master's current role is: LEADER
I20260925 04:48:43.180737 393 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:48:43.182606 22059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45f90b19124d4a039140d7bdc7cb7079 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "45f90b19124d4a039140d7bdc7cb7079" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f90b19124d4a039140d7bdc7cb7079" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39729 } } }
I20260925 04:48:43.182726 22059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45f90b19124d4a039140d7bdc7cb7079 [sys.catalog]: This master's current role is: LEADER
I20260925 04:48:43.183028 22073 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:48:43.186151 22073 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:48:43.192401 22056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:43.182861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53157 (local address 127.0.98.126:39729)
0925 04:48:43.182982 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:43.182986 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:43.183352 (+ 366us) server_negotiation.cc:407] Connection header received
0925 04:48:43.183468 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:43.183470 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:43.183512 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:43.183579 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:43.184448 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:43.185242 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:43.186415 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:43.186772 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:43.186886 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:43.188608 (+ 1722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:43.188612 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:43.188615 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:48:43.188639 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:43.190202 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:43.190890 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:43.190894 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:43.190895 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:43.190985 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:43.191395 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:43.191399 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:43.191401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:43.191632 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:43.191781 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:43.192138 (+ 357us) server_negotiation.cc:299] Negotiation successful
0925 04:48:43.192240 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 04:48:43.192473 22073 catalog_manager.cc:1347] Generated new cluster ID: 0ed9ed5cda89454a90ea1086729a5b39
I20260925 04:48:43.192517 22073 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:48:43.207243 22073 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:48:43.208206 22073 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:48:43.214093 22073 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 45f90b19124d4a039140d7bdc7cb7079: Generated new TSK 0
I20260925 04:48:43.214852 22073 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:48:43.221477 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 21985
W20260925 04:48:43.227012 21970 connection.cc:535] client connection to 127.0.98.126:39729 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.126:39729): Connection reset by peer (error 104)
2026-09-25T04:48:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1463 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790311449873301-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:48:43 dist-test-slave-kghz krb5kdc[22084](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:48:43 dist-test-slave-kghz krb5kdc[22084](info): set up 2 sockets
Sep 25 04:48:43 dist-test-slave-kghz krb5kdc[22084](info): commencing operation
krb5kdc: starting...
W20260925 04:48:44.267557 393 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790311449873301-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790311449873301-393-0/krb5kdc/test-user.keytab.
Sep 25 04:48:44 dist-test-slave-kghz krb5kdc[22084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311724, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:48:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:48:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260925 04:48:44.397637 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790311449873301-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790311449873301-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790311449873301-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:37495
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:33171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:37495
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:48:44.508234 22097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:48:44.508510 22097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:48:44.508554 22097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:48:44.512768 22097 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:48:44.512867 22097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:48:44.512888 22097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:48:44.512907 22097 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:48:44.512921 22097 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:48:44.518399 22097 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:33171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790311449873301-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790311449873301-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:37495
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:37495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22097
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790311449873301-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:48:44.519506 22097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:48:44.520624 22097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:48:44.529734 22102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:48:44.529965 22097 server_base.cc:1034] running on GCE node
W20260925 04:48:44.529712 22105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:48:44.529706 22103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:48:44.530774 22097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:48:44.531919 22097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:48:44.533109 22097 hybrid_clock.cc:648] HybridClock initialized: now 1790311724533080 us; error 34 us; skew 500 ppm
Sep 25 04:48:44 dist-test-slave-kghz krb5kdc[22084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311724, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:48:44.537026 22097 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260925 04:48:44.537997 22097 webserver.cc:460] Webserver started at http://127.0.98.126:34607/ using document root <none> and password file <none>
I20260925 04:48:44.538640 22097 fs_manager.cc:359] Metadata directory not provided
I20260925 04:48:44.538734 22097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:48:44.538981 22097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:48:44.541034 22097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790311449873301-393-0/minicluster-data/master-0/data/instance:
uuid: "4db3489cc6e545a4bd8221d9e8cdd22e"
format_stamp: "Formatted at 2026-09-25 04:48:44 on dist-test-slave-kghz"
server_key: "a0a1600bb1f69b44667e70559e25e8dd"
server_key_iv: "cdfc2ccaf3704dac804827e3ab0e6088"
server_key_version: "encryptionkey@0"
I20260925 04:48:44.541613 22097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790311449873301-393-0/minicluster-data/master-0/wal/instance:
uuid: "4db3489cc6e545a4bd8221d9e8cdd22e"
format_stamp: "Formatted at 2026-09-25 04:48:44 on dist-test-slave-kghz"
server_key: "a0a1600bb1f69b44667e70559e25e8dd"
server_key_iv: "cdfc2ccaf3704dac804827e3ab0e6088"
server_key_version: "encryptionkey@0"
I20260925 04:48:44.545364 22097 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 04:48:44.547868 22112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:48:44.549108 22097 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 04:48:44.549256 22097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790311449873301-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790311449873301-393-0/minicluster-data/master-0/wal
uuid: "4db3489cc6e545a4bd8221d9e8cdd22e"
format_stamp: "Formatted at 2026-09-25 04:48:44 on dist-test-slave-kghz"
server_key: "a0a1600bb1f69b44667e70559e25e8dd"
server_key_iv: "cdfc2ccaf3704dac804827e3ab0e6088"
server_key_version: "encryptionkey@0"
I20260925 04:48:44.549381 22097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790311449873301-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790311449873301-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790311449873301-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:48:44.558452 22097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:48:44.561435 22097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:48:44.561655 22097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:48:44.569260 22097 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:37495
I20260925 04:48:44.569278 22164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:37495 every 8 connection(s)
I20260925 04:48:44.570304 22097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
I20260925 04:48:44.573256 22165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:48:44.574376 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 22097
I20260925 04:48:44.574512 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790311449873301-393-0/minicluster-data/master-0/wal/instance
I20260925 04:48:44.574837 393 external_mini_cluster.cc:1428] Setting key 8a8b4a219bdcb16e4c545a7fb40fc2f7
I20260925 04:48:44.579571 22165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4db3489cc6e545a4bd8221d9e8cdd22e: Bootstrap starting.
I20260925 04:48:44.581952 22165 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4db3489cc6e545a4bd8221d9e8cdd22e: Neither blocks nor log segments found. Creating new log.
I20260925 04:48:44.582798 22165 log.cc:824] T 00000000000000000000000000000000 P 4db3489cc6e545a4bd8221d9e8cdd22e: Log is configured to *not* fsync() on all Append() calls
I20260925 04:48:44.584441 22168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:44.576307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47163 (local address 127.0.98.126:37495)
0925 04:48:44.576781 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:44.576796 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:44.576833 (+ 37us) server_negotiation.cc:407] Connection header received
0925 04:48:44.577485 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:44.577501 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:44.577869 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:44.578218 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:44.578671 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:44.579692 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:44.580620 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:44.581126 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:44.581294 (+ 168us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:44.582462 (+ 1168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:44.582467 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:44.582481 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:48:44.582947 (+ 466us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:44.583021 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:48:44.583045 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:44.583191 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:44.583367 (+ 176us) server_negotiation.cc:299] Negotiation successful
0925 04:48:44.583510 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":137,"threads_started":1}
I20260925 04:48:44.585064 22165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4db3489cc6e545a4bd8221d9e8cdd22e: No bootstrap required, opened a new log
I20260925 04:48:44.587473 393 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:48:44.589048 22165 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4db3489cc6e545a4bd8221d9e8cdd22e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4db3489cc6e545a4bd8221d9e8cdd22e" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 37495 } }
I20260925 04:48:44.589361 22165 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4db3489cc6e545a4bd8221d9e8cdd22e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:48:44.589455 22165 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4db3489cc6e545a4bd8221d9e8cdd22e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4db3489cc6e545a4bd8221d9e8cdd22e, State: Initialized, Role: FOLLOWER
I20260925 04:48:44.589913 22165 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4db3489cc6e545a4bd8221d9e8cdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4db3489cc6e545a4bd8221d9e8cdd22e" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 37495 } }
I20260925 04:48:44.590081 22165 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4db3489cc6e545a4bd8221d9e8cdd22e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:48:44.590155 22165 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4db3489cc6e545a4bd8221d9e8cdd22e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:48:44.590318 22165 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4db3489cc6e545a4bd8221d9e8cdd22e [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:48:44.591500 22165 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4db3489cc6e545a4bd8221d9e8cdd22e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4db3489cc6e545a4bd8221d9e8cdd22e" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 37495 } }
I20260925 04:48:44.591856 22165 leader_election.cc:302] T 00000000000000000000000000000000 P 4db3489cc6e545a4bd8221d9e8cdd22e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4db3489cc6e545a4bd8221d9e8cdd22e; no voters:
I20260925 04:48:44.592228 22165 leader_election.cc:288] T 00000000000000000000000000000000 P 4db3489cc6e545a4bd8221d9e8cdd22e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:48:44.592278 22175 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4db3489cc6e545a4bd8221d9e8cdd22e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:48:44.592532 22175 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4db3489cc6e545a4bd8221d9e8cdd22e [term 1 LEADER]: Becoming Leader. State: Replica: 4db3489cc6e545a4bd8221d9e8cdd22e, State: Running, Role: LEADER
I20260925 04:48:44.592913 22175 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4db3489cc6e545a4bd8221d9e8cdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4db3489cc6e545a4bd8221d9e8cdd22e" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 37495 } }
I20260925 04:48:44.593593 22165 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4db3489cc6e545a4bd8221d9e8cdd22e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:48:44.594421 22168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:44.589559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36061 (local address 127.0.98.126:37495)
0925 04:48:44.589677 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:44.589681 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:44.589882 (+ 201us) server_negotiation.cc:407] Connection header received
0925 04:48:44.590058 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:44.590062 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:44.590111 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:44.590201 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:44.590503 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:44.591252 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:44.592297 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:44.592674 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:44.593225 (+ 551us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:44.593778 (+ 553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:44.593781 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:44.593783 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:48:44.593794 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:44.593817 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:48:44.593827 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:44.593932 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:44.594216 (+ 284us) server_negotiation.cc:299] Negotiation successful
0925 04:48:44.594284 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 04:48:44.594830 22176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4db3489cc6e545a4bd8221d9e8cdd22e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4db3489cc6e545a4bd8221d9e8cdd22e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4db3489cc6e545a4bd8221d9e8cdd22e" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 37495 } } }
I20260925 04:48:44.595026 22176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4db3489cc6e545a4bd8221d9e8cdd22e [sys.catalog]: This master's current role is: LEADER
I20260925 04:48:44.595445 22177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4db3489cc6e545a4bd8221d9e8cdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4db3489cc6e545a4bd8221d9e8cdd22e. Latest consensus state: current_term: 1 leader_uuid: "4db3489cc6e545a4bd8221d9e8cdd22e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4db3489cc6e545a4bd8221d9e8cdd22e" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 37495 } } }
I20260925 04:48:44.595614 22177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4db3489cc6e545a4bd8221d9e8cdd22e [sys.catalog]: This master's current role is: LEADER
W20260925 04:48:44.599313 22190 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4db3489cc6e545a4bd8221d9e8cdd22e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 04:48:44.599466 22190 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 04:48:44.599570 22185 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:48:44.600294 22185 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:48:44.605726 22185 catalog_manager.cc:1347] Generated new cluster ID: 41c3a4c1065d4582b916f7ef42291294
I20260925 04:48:44.605806 22185 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:48:44.625022 22185 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:48:44.626003 22185 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:48:44.629817 22185 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4db3489cc6e545a4bd8221d9e8cdd22e: Generated new TSK 0
I20260925 04:48:44.630609 22185 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:48:44.640023 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 22097
W20260925 04:48:44.647004 22082 connection.cc:535] client connection to 127.0.98.126:37495 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.126:37495): Connection reset by peer (error 104)
2026-09-25T04:48:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1420 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790311449873301-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:48:44 dist-test-slave-kghz krb5kdc[22196](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:48:44 dist-test-slave-kghz krb5kdc[22196](info): set up 2 sockets
Sep 25 04:48:44 dist-test-slave-kghz krb5kdc[22196](info): commencing operation
krb5kdc: starting...
W20260925 04:48:45.695154 393 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790311449873301-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790311449873301-393-0/krb5kdc/test-user.keytab.
Sep 25 04:48:45 dist-test-slave-kghz krb5kdc[22196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:48:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:48:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260925 04:48:45.821686 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790311449873301-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790311449873301-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790311449873301-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:33167
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:32829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:33167
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:48:45.938400 22209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:48:45.938688 22209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:48:45.938730 22209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:48:45.942759 22209 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:48:45.942826 22209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:48:45.942845 22209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:48:45.942862 22209 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:48:45.942878 22209 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:48:45.948182 22209 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:32829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790311449873301-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790311449873301-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:33167
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:33167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22209
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790311449873301-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:48:45.949224 22209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:48:45.950205 22209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:48:45.959326 22215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:48:45.959322 22217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:48:45.959435 22209 server_base.cc:1034] running on GCE node
W20260925 04:48:45.959328 22214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:48:45.960170 22209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:48:45.961303 22209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:48:45.962502 22209 hybrid_clock.cc:648] HybridClock initialized: now 1790311725962485 us; error 28 us; skew 500 ppm
Sep 25 04:48:45 dist-test-slave-kghz krb5kdc[22196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311725, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:48:45.965230 22209 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260925 04:48:45.966176 22209 webserver.cc:460] Webserver started at http://127.0.98.126:45429/ using document root <none> and password file <none>
I20260925 04:48:45.966765 22209 fs_manager.cc:359] Metadata directory not provided
I20260925 04:48:45.966858 22209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:48:45.967095 22209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:48:45.969146 22209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790311449873301-393-0/minicluster-data/master-0/data/instance:
uuid: "b2ebd7dbb3bb4522b76de27fb0b9aed0"
format_stamp: "Formatted at 2026-09-25 04:48:45 on dist-test-slave-kghz"
server_key: "c32fa91b1266607a6a4da182f8c5b0ef"
server_key_iv: "c9451835bcceeab9822f07d3307ad409"
server_key_version: "encryptionkey@0"
I20260925 04:48:45.969730 22209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790311449873301-393-0/minicluster-data/master-0/wal/instance:
uuid: "b2ebd7dbb3bb4522b76de27fb0b9aed0"
format_stamp: "Formatted at 2026-09-25 04:48:45 on dist-test-slave-kghz"
server_key: "c32fa91b1266607a6a4da182f8c5b0ef"
server_key_iv: "c9451835bcceeab9822f07d3307ad409"
server_key_version: "encryptionkey@0"
I20260925 04:48:45.973701 22209 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260925 04:48:45.976303 22224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:48:45.977576 22209 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 04:48:45.977716 22209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790311449873301-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790311449873301-393-0/minicluster-data/master-0/wal
uuid: "b2ebd7dbb3bb4522b76de27fb0b9aed0"
format_stamp: "Formatted at 2026-09-25 04:48:45 on dist-test-slave-kghz"
server_key: "c32fa91b1266607a6a4da182f8c5b0ef"
server_key_iv: "c9451835bcceeab9822f07d3307ad409"
server_key_version: "encryptionkey@0"
I20260925 04:48:45.977846 22209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790311449873301-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790311449873301-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790311449873301-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:48:45.982597 22209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:48:45.985769 22209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:48:45.985971 22209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:48:45.993369 22209 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:33167
I20260925 04:48:45.993397 22276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:33167 every 8 connection(s)
I20260925 04:48:45.994405 22209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
I20260925 04:48:45.997105 22277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:48:46.000542 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 22209
I20260925 04:48:46.000684 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790311449873301-393-0/minicluster-data/master-0/wal/instance
I20260925 04:48:46.000972 393 external_mini_cluster.cc:1428] Setting key e9058331384c4a5040678ba8d2ef9ac5
I20260925 04:48:46.003152 22277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2ebd7dbb3bb4522b76de27fb0b9aed0: Bootstrap starting.
I20260925 04:48:46.005539 22277 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2ebd7dbb3bb4522b76de27fb0b9aed0: Neither blocks nor log segments found. Creating new log.
I20260925 04:48:46.006343 22277 log.cc:824] T 00000000000000000000000000000000 P b2ebd7dbb3bb4522b76de27fb0b9aed0: Log is configured to *not* fsync() on all Append() calls
I20260925 04:48:46.006757 22280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:46.002496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58017 (local address 127.0.98.126:33167)
0925 04:48:46.002941 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:46.002956 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:46.002994 (+ 38us) server_negotiation.cc:407] Connection header received
0925 04:48:46.003708 (+ 714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:46.003728 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:46.004024 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:46.004365 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:46.005304 (+ 939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:46.005309 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:46.005321 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:48:46.005332 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:46.005385 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:48:46.005411 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:46.005526 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:46.005642 (+ 116us) server_negotiation.cc:299] Negotiation successful
0925 04:48:46.005923 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":143,"threads_started":1}
I20260925 04:48:46.008438 22277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2ebd7dbb3bb4522b76de27fb0b9aed0: No bootstrap required, opened a new log
I20260925 04:48:46.009784 393 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:48:46.012509 22277 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2ebd7dbb3bb4522b76de27fb0b9aed0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2ebd7dbb3bb4522b76de27fb0b9aed0" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33167 } }
I20260925 04:48:46.012825 22277 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2ebd7dbb3bb4522b76de27fb0b9aed0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:48:46.012913 22277 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2ebd7dbb3bb4522b76de27fb0b9aed0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2ebd7dbb3bb4522b76de27fb0b9aed0, State: Initialized, Role: FOLLOWER
I20260925 04:48:46.013474 22277 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2ebd7dbb3bb4522b76de27fb0b9aed0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2ebd7dbb3bb4522b76de27fb0b9aed0" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33167 } }
I20260925 04:48:46.013665 22277 raft_consensus.cc:388] T 00000000000000000000000000000000 P b2ebd7dbb3bb4522b76de27fb0b9aed0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:48:46.013783 22277 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2ebd7dbb3bb4522b76de27fb0b9aed0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:48:46.013875 22280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:46.012003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37425 (local address 127.0.98.126:33167)
0925 04:48:46.012138 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:46.012145 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:46.012279 (+ 134us) server_negotiation.cc:407] Connection header received
0925 04:48:46.012417 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:46.012419 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:46.012449 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:46.012519 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:46.013439 (+ 920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:46.013441 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:46.013443 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:48:46.013453 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:46.013476 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:48:46.013486 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:46.013565 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:46.013670 (+ 105us) server_negotiation.cc:299] Negotiation successful
0925 04:48:46.013708 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 04:48:46.013907 22277 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2ebd7dbb3bb4522b76de27fb0b9aed0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:48:46.015050 22277 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2ebd7dbb3bb4522b76de27fb0b9aed0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2ebd7dbb3bb4522b76de27fb0b9aed0" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33167 } }
I20260925 04:48:46.015401 22277 leader_election.cc:302] T 00000000000000000000000000000000 P b2ebd7dbb3bb4522b76de27fb0b9aed0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b2ebd7dbb3bb4522b76de27fb0b9aed0; no voters:
I20260925 04:48:46.015699 22277 leader_election.cc:288] T 00000000000000000000000000000000 P b2ebd7dbb3bb4522b76de27fb0b9aed0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:48:46.015831 22287 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2ebd7dbb3bb4522b76de27fb0b9aed0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:48:46.016067 22287 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2ebd7dbb3bb4522b76de27fb0b9aed0 [term 1 LEADER]: Becoming Leader. State: Replica: b2ebd7dbb3bb4522b76de27fb0b9aed0, State: Running, Role: LEADER
I20260925 04:48:46.016420 22287 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2ebd7dbb3bb4522b76de27fb0b9aed0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2ebd7dbb3bb4522b76de27fb0b9aed0" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33167 } }
I20260925 04:48:46.016911 22277 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2ebd7dbb3bb4522b76de27fb0b9aed0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:48:46.018064 22288 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2ebd7dbb3bb4522b76de27fb0b9aed0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b2ebd7dbb3bb4522b76de27fb0b9aed0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2ebd7dbb3bb4522b76de27fb0b9aed0" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33167 } } }
I20260925 04:48:46.018198 22288 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2ebd7dbb3bb4522b76de27fb0b9aed0 [sys.catalog]: This master's current role is: LEADER
I20260925 04:48:46.018399 22289 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2ebd7dbb3bb4522b76de27fb0b9aed0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2ebd7dbb3bb4522b76de27fb0b9aed0. Latest consensus state: current_term: 1 leader_uuid: "b2ebd7dbb3bb4522b76de27fb0b9aed0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2ebd7dbb3bb4522b76de27fb0b9aed0" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33167 } } }
I20260925 04:48:46.018473 22289 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2ebd7dbb3bb4522b76de27fb0b9aed0 [sys.catalog]: This master's current role is: LEADER
I20260925 04:48:46.019460 22298 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:48:46.022547 22298 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:48:46.027827 22298 catalog_manager.cc:1347] Generated new cluster ID: e88467c92081469a9233c5a1c13fe45d
I20260925 04:48:46.027944 22298 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:48:46.040570 22298 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:48:46.040935 22298 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:48:46.047417 22298 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b2ebd7dbb3bb4522b76de27fb0b9aed0: Generated new TSK 0
I20260925 04:48:46.048133 22298 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:48:46.059692 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 22209
2026-09-25T04:48:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1418 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790311449873301-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:48:46 dist-test-slave-kghz krb5kdc[22308](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:48:46 dist-test-slave-kghz krb5kdc[22308](info): set up 2 sockets
Sep 25 04:48:46 dist-test-slave-kghz krb5kdc[22308](info): commencing operation
krb5kdc: starting...
W20260925 04:48:47.106557 393 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790311449873301-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790311449873301-393-0/krb5kdc/test-user.keytab.
Sep 25 04:48:47 dist-test-slave-kghz krb5kdc[22308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311727, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:48:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:48:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260925 04:48:47.233660 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790311449873301-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790311449873301-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790311449873301-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:36743
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:41323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:36743
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:48:47.347466 22321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:48:47.347741 22321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:48:47.347784 22321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:48:47.351750 22321 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:48:47.351809 22321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:48:47.351827 22321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:48:47.351843 22321 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:48:47.351858 22321 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:48:47.357064 22321 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:41323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790311449873301-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790311449873301-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:36743
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:36743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22321
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790311449873301-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:48:47.358079 22321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:48:47.359143 22321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:48:47.367772 22326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:48:47.367764 22329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:48:47.367764 22327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:48:47.368065 22321 server_base.cc:1034] running on GCE node
I20260925 04:48:47.368687 22321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:48:47.369900 22321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:48:47.371098 22321 hybrid_clock.cc:648] HybridClock initialized: now 1790311727371084 us; error 29 us; skew 500 ppm
Sep 25 04:48:47 dist-test-slave-kghz krb5kdc[22308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311727, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:48:47.374014 22321 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260925 04:48:47.374970 22321 webserver.cc:460] Webserver started at http://127.0.98.126:35385/ using document root <none> and password file <none>
I20260925 04:48:47.375509 22321 fs_manager.cc:359] Metadata directory not provided
I20260925 04:48:47.375568 22321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:48:47.375733 22321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:48:47.377849 22321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790311449873301-393-0/minicluster-data/master-0/data/instance:
uuid: "3ba2c5c5f50248dda8e328a1440bf5a7"
format_stamp: "Formatted at 2026-09-25 04:48:47 on dist-test-slave-kghz"
server_key: "09e1a55f0db8b2202bbcd5adbe22f11b"
server_key_iv: "a5543166c3d3041afb228c20ec77e0f0"
server_key_version: "encryptionkey@0"
I20260925 04:48:47.378767 22321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790311449873301-393-0/minicluster-data/master-0/wal/instance:
uuid: "3ba2c5c5f50248dda8e328a1440bf5a7"
format_stamp: "Formatted at 2026-09-25 04:48:47 on dist-test-slave-kghz"
server_key: "09e1a55f0db8b2202bbcd5adbe22f11b"
server_key_iv: "a5543166c3d3041afb228c20ec77e0f0"
server_key_version: "encryptionkey@0"
I20260925 04:48:47.382884 22321 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 04:48:47.385514 22336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:48:47.386891 22321 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:48:47.387034 22321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790311449873301-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790311449873301-393-0/minicluster-data/master-0/wal
uuid: "3ba2c5c5f50248dda8e328a1440bf5a7"
format_stamp: "Formatted at 2026-09-25 04:48:47 on dist-test-slave-kghz"
server_key: "09e1a55f0db8b2202bbcd5adbe22f11b"
server_key_iv: "a5543166c3d3041afb228c20ec77e0f0"
server_key_version: "encryptionkey@0"
I20260925 04:48:47.387151 22321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790311449873301-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790311449873301-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790311449873301-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:48:47.404073 22321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:48:47.407538 22321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:48:47.407775 22321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:48:47.415122 22321 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:36743
I20260925 04:48:47.415138 22388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:36743 every 8 connection(s)
I20260925 04:48:47.416088 22321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
I20260925 04:48:47.418910 22389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:48:47.420173 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 22321
I20260925 04:48:47.420323 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790311449873301-393-0/minicluster-data/master-0/wal/instance
I20260925 04:48:47.420604 393 external_mini_cluster.cc:1428] Setting key 23cb8f752792980a0196ff879408db31
I20260925 04:48:47.425645 22389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ba2c5c5f50248dda8e328a1440bf5a7: Bootstrap starting.
I20260925 04:48:47.428261 22389 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ba2c5c5f50248dda8e328a1440bf5a7: Neither blocks nor log segments found. Creating new log.
I20260925 04:48:47.429112 22389 log.cc:824] T 00000000000000000000000000000000 P 3ba2c5c5f50248dda8e328a1440bf5a7: Log is configured to *not* fsync() on all Append() calls
Sep 25 04:48:47 dist-test-slave-kghz krb5kdc[22308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311727, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:48:47.431524 22389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ba2c5c5f50248dda8e328a1440bf5a7: No bootstrap required, opened a new log
I20260925 04:48:47.435894 22389 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ba2c5c5f50248dda8e328a1440bf5a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ba2c5c5f50248dda8e328a1440bf5a7" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 36743 } }
I20260925 04:48:47.436156 22389 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ba2c5c5f50248dda8e328a1440bf5a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:48:47.436265 22389 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ba2c5c5f50248dda8e328a1440bf5a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ba2c5c5f50248dda8e328a1440bf5a7, State: Initialized, Role: FOLLOWER
I20260925 04:48:47.436597 22392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:47.422037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44275 (local address 127.0.98.126:36743)
0925 04:48:47.422473 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:47.422492 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:47.422535 (+ 43us) server_negotiation.cc:407] Connection header received
0925 04:48:47.423341 (+ 806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:47.423364 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:47.423795 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:47.424168 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:47.424855 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:47.426341 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:47.427218 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:47.427887 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:47.428044 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:47.431183 (+ 3139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:47.431188 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:47.431200 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:48:47.431231 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:47.433206 (+ 1975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:47.433827 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:47.433833 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:47.433840 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:47.433953 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:47.434383 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:47.434386 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:47.434387 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:47.434786 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:47.434968 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:47.435266 (+ 298us) server_negotiation.cc:299] Negotiation successful
0925 04:48:47.435624 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":143,"threads_started":1}
I20260925 04:48:47.436734 22389 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ba2c5c5f50248dda8e328a1440bf5a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ba2c5c5f50248dda8e328a1440bf5a7" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 36743 } }
I20260925 04:48:47.436919 22389 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3ba2c5c5f50248dda8e328a1440bf5a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:48:47.436990 22389 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ba2c5c5f50248dda8e328a1440bf5a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:48:47.437125 22389 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ba2c5c5f50248dda8e328a1440bf5a7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:48:47.438092 22389 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ba2c5c5f50248dda8e328a1440bf5a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ba2c5c5f50248dda8e328a1440bf5a7" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 36743 } }
I20260925 04:48:47.438465 22389 leader_election.cc:302] T 00000000000000000000000000000000 P 3ba2c5c5f50248dda8e328a1440bf5a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ba2c5c5f50248dda8e328a1440bf5a7; no voters:
I20260925 04:48:47.438820 22389 leader_election.cc:288] T 00000000000000000000000000000000 P 3ba2c5c5f50248dda8e328a1440bf5a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:48:47.438898 22394 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ba2c5c5f50248dda8e328a1440bf5a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:48:47.439114 22394 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ba2c5c5f50248dda8e328a1440bf5a7 [term 1 LEADER]: Becoming Leader. State: Replica: 3ba2c5c5f50248dda8e328a1440bf5a7, State: Running, Role: LEADER
I20260925 04:48:47.439473 22394 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ba2c5c5f50248dda8e328a1440bf5a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ba2c5c5f50248dda8e328a1440bf5a7" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 36743 } }
I20260925 04:48:47.439857 393 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:48:47.440164 22389 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ba2c5c5f50248dda8e328a1440bf5a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:48:47.441130 22395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ba2c5c5f50248dda8e328a1440bf5a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ba2c5c5f50248dda8e328a1440bf5a7. Latest consensus state: current_term: 1 leader_uuid: "3ba2c5c5f50248dda8e328a1440bf5a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ba2c5c5f50248dda8e328a1440bf5a7" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 36743 } } }
I20260925 04:48:47.441252 22395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ba2c5c5f50248dda8e328a1440bf5a7 [sys.catalog]: This master's current role is: LEADER
I20260925 04:48:47.441469 22396 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ba2c5c5f50248dda8e328a1440bf5a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ba2c5c5f50248dda8e328a1440bf5a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ba2c5c5f50248dda8e328a1440bf5a7" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 36743 } } }
I20260925 04:48:47.441553 22396 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ba2c5c5f50248dda8e328a1440bf5a7 [sys.catalog]: This master's current role is: LEADER
I20260925 04:48:47.441807 22407 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:48:47.445082 22407 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:48:47.451270 22407 catalog_manager.cc:1347] Generated new cluster ID: 0b9455dcc9dc403cb3d6be4aecb0710f
I20260925 04:48:47.451340 22407 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:48:47.453341 22392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:47.442210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41449 (local address 127.0.98.126:36743)
0925 04:48:47.442363 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:47.442370 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:47.443129 (+ 759us) server_negotiation.cc:407] Connection header received
0925 04:48:47.443266 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:47.443270 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:47.443320 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:47.443394 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:47.444276 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:47.445145 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:47.446241 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:47.446615 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:47.446745 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:47.449920 (+ 3175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:47.449926 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:47.449929 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:48:47.449956 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:47.451680 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:47.452110 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:47.452115 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:47.452117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:47.452197 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:47.452492 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:47.452495 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:47.452497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:47.452674 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:47.452815 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:47.453054 (+ 239us) server_negotiation.cc:299] Negotiation successful
0925 04:48:47.453175 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 04:48:47.467140 22407 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:48:47.468096 22407 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:48:47.472301 22407 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3ba2c5c5f50248dda8e328a1440bf5a7: Generated new TSK 0
I20260925 04:48:47.473021 22407 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:48:47.481757 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 22321
2026-09-25T04:48:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1421 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790311449873301-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:48:47 dist-test-slave-kghz krb5kdc[22420](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:48:47 dist-test-slave-kghz krb5kdc[22420](info): set up 2 sockets
Sep 25 04:48:47 dist-test-slave-kghz krb5kdc[22420](info): commencing operation
krb5kdc: starting...
W20260925 04:48:48.566623 393 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790311449873301-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790311449873301-393-0/krb5kdc/test-user.keytab.
Sep 25 04:48:48 dist-test-slave-kghz krb5kdc[22420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311728, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:48:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:48:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260925 04:48:48.697964 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790311449873301-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790311449873301-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790311449873301-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:44387
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:35813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:44387
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:48:48.810278 22433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:48:48.810593 22433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:48:48.810678 22433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:48:48.814781 22433 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:48:48.814872 22433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:48:48.814911 22433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:48:48.814975 22433 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:48:48.815012 22433 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:48:48.820416 22433 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:35813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790311449873301-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790311449873301-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:44387
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:44387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790311449873301-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:48:48.821548 22433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:48:48.822633 22433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:48:48.831583 22439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:48:48.831645 22433 server_base.cc:1034] running on GCE node
W20260925 04:48:48.831573 22438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:48:48.831540 22441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:48:48.832291 22433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:48:48.833458 22433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:48:48.834630 22433 hybrid_clock.cc:648] HybridClock initialized: now 1790311728834616 us; error 27 us; skew 500 ppm
Sep 25 04:48:48 dist-test-slave-kghz krb5kdc[22420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311728, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:48:48.837644 22433 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260925 04:48:48.838716 22433 webserver.cc:460] Webserver started at http://127.0.98.126:33411/ using document root <none> and password file <none>
I20260925 04:48:48.839300 22433 fs_manager.cc:359] Metadata directory not provided
I20260925 04:48:48.839368 22433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:48:48.839587 22433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:48:48.841675 22433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790311449873301-393-0/minicluster-data/master-0/data/instance:
uuid: "6f6f6e178c00493e99ed76d0414b5dc6"
format_stamp: "Formatted at 2026-09-25 04:48:48 on dist-test-slave-kghz"
server_key: "d440fa3185b203a373828055c474f2e2"
server_key_iv: "2e41aef69a3e96a0cc4799efaeabc426"
server_key_version: "encryptionkey@0"
I20260925 04:48:48.842293 22433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790311449873301-393-0/minicluster-data/master-0/wal/instance:
uuid: "6f6f6e178c00493e99ed76d0414b5dc6"
format_stamp: "Formatted at 2026-09-25 04:48:48 on dist-test-slave-kghz"
server_key: "d440fa3185b203a373828055c474f2e2"
server_key_iv: "2e41aef69a3e96a0cc4799efaeabc426"
server_key_version: "encryptionkey@0"
I20260925 04:48:48.846052 22433 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:48:48.848570 22448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:48:48.849915 22433 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 04:48:48.850060 22433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790311449873301-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790311449873301-393-0/minicluster-data/master-0/wal
uuid: "6f6f6e178c00493e99ed76d0414b5dc6"
format_stamp: "Formatted at 2026-09-25 04:48:48 on dist-test-slave-kghz"
server_key: "d440fa3185b203a373828055c474f2e2"
server_key_iv: "2e41aef69a3e96a0cc4799efaeabc426"
server_key_version: "encryptionkey@0"
I20260925 04:48:48.850183 22433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790311449873301-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790311449873301-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790311449873301-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:48:48.866134 22433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:48:48.869588 22433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:48:48.869807 22433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:48:48.877290 22433 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:44387
I20260925 04:48:48.877308 22500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:44387 every 8 connection(s)
I20260925 04:48:48.878280 22433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
I20260925 04:48:48.881084 22501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:48:48.885039 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 22433
I20260925 04:48:48.885200 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790311449873301-393-0/minicluster-data/master-0/wal/instance
I20260925 04:48:48.885524 393 external_mini_cluster.cc:1428] Setting key fe6ad01baf98298959a8aa7fee5ed8c8
I20260925 04:48:48.887377 22501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f6f6e178c00493e99ed76d0414b5dc6: Bootstrap starting.
I20260925 04:48:48.889796 22501 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f6f6e178c00493e99ed76d0414b5dc6: Neither blocks nor log segments found. Creating new log.
I20260925 04:48:48.890720 22501 log.cc:824] T 00000000000000000000000000000000 P 6f6f6e178c00493e99ed76d0414b5dc6: Log is configured to *not* fsync() on all Append() calls
I20260925 04:48:48.893003 22501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f6f6e178c00493e99ed76d0414b5dc6: No bootstrap required, opened a new log
I20260925 04:48:48.895125 22504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:48.887024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42583 (local address 127.0.98.126:44387)
0925 04:48:48.887450 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:48.887473 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:48.887521 (+ 48us) server_negotiation.cc:407] Connection header received
0925 04:48:48.888189 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:48.888207 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:48.888555 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:48.888846 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:48.889169 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:48.890325 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:48.891453 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:48.891994 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:48.892124 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:48.893009 (+ 885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:48.893014 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:48.893026 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:48:48.893467 (+ 441us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:48.893522 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:48:48.893546 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:48.893626 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:48.893833 (+ 207us) server_negotiation.cc:299] Negotiation successful
0925 04:48:48.894098 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":143,"threads_started":1}
I20260925 04:48:48.896283 22501 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f6f6e178c00493e99ed76d0414b5dc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f6f6e178c00493e99ed76d0414b5dc6" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 44387 } }
I20260925 04:48:48.896504 22501 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f6f6e178c00493e99ed76d0414b5dc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:48:48.896565 22501 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f6f6e178c00493e99ed76d0414b5dc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f6f6e178c00493e99ed76d0414b5dc6, State: Initialized, Role: FOLLOWER
I20260925 04:48:48.897033 22501 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f6f6e178c00493e99ed76d0414b5dc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f6f6e178c00493e99ed76d0414b5dc6" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 44387 } }
I20260925 04:48:48.897207 22501 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f6f6e178c00493e99ed76d0414b5dc6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:48:48.897274 22501 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f6f6e178c00493e99ed76d0414b5dc6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:48:48.897406 22501 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f6f6e178c00493e99ed76d0414b5dc6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:48:48.898074 393 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:48:48.898561 22501 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f6f6e178c00493e99ed76d0414b5dc6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f6f6e178c00493e99ed76d0414b5dc6" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 44387 } }
I20260925 04:48:48.898970 22501 leader_election.cc:302] T 00000000000000000000000000000000 P 6f6f6e178c00493e99ed76d0414b5dc6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f6f6e178c00493e99ed76d0414b5dc6; no voters:
I20260925 04:48:48.899317 22501 leader_election.cc:288] T 00000000000000000000000000000000 P 6f6f6e178c00493e99ed76d0414b5dc6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:48:48.899444 22506 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f6f6e178c00493e99ed76d0414b5dc6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:48:48.899701 22506 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f6f6e178c00493e99ed76d0414b5dc6 [term 1 LEADER]: Becoming Leader. State: Replica: 6f6f6e178c00493e99ed76d0414b5dc6, State: Running, Role: LEADER
I20260925 04:48:48.900074 22506 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f6f6e178c00493e99ed76d0414b5dc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f6f6e178c00493e99ed76d0414b5dc6" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 44387 } }
I20260925 04:48:48.900699 22501 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f6f6e178c00493e99ed76d0414b5dc6 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:48:48.901999 22512 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f6f6e178c00493e99ed76d0414b5dc6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f6f6e178c00493e99ed76d0414b5dc6. Latest consensus state: current_term: 1 leader_uuid: "6f6f6e178c00493e99ed76d0414b5dc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f6f6e178c00493e99ed76d0414b5dc6" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 44387 } } }
I20260925 04:48:48.902006 22509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f6f6e178c00493e99ed76d0414b5dc6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f6f6e178c00493e99ed76d0414b5dc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f6f6e178c00493e99ed76d0414b5dc6" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 44387 } } }
I20260925 04:48:48.902158 22509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f6f6e178c00493e99ed76d0414b5dc6 [sys.catalog]: This master's current role is: LEADER
I20260925 04:48:48.902155 22512 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f6f6e178c00493e99ed76d0414b5dc6 [sys.catalog]: This master's current role is: LEADER
I20260925 04:48:48.902577 22520 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:48:48.905866 22520 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:48:48.905865 22504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:48.900348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37571 (local address 127.0.98.126:44387)
0925 04:48:48.900459 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:48.900464 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:48.900950 (+ 486us) server_negotiation.cc:407] Connection header received
0925 04:48:48.901089 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:48.901094 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:48.901142 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:48.901222 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:48.901579 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:48.902425 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:48.903696 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:48.904026 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:48.904137 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:48.905193 (+ 1056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:48.905197 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:48.905200 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:48:48.905225 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:48.905254 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:48:48.905267 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:48.905331 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:48.905673 (+ 342us) server_negotiation.cc:299] Negotiation successful
0925 04:48:48.905734 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 04:48:48.918690 22520 catalog_manager.cc:1347] Generated new cluster ID: 1a70017730ba49f4a863a246e798ad1a
I20260925 04:48:48.918833 22520 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:48:48.940914 22520 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:48:48.942150 22520 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:48:48.951424 22520 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f6f6e178c00493e99ed76d0414b5dc6: Generated new TSK 0
I20260925 04:48:48.952328 22520 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:48:48.969089 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 22433
2026-09-25T04:48:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1487 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790311449873301-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:48:49 dist-test-slave-kghz krb5kdc[22532](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:48:49 dist-test-slave-kghz krb5kdc[22532](info): set up 2 sockets
Sep 25 04:48:49 dist-test-slave-kghz krb5kdc[22532](info): commencing operation
krb5kdc: starting...
W20260925 04:48:50.006704 393 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790311449873301-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790311449873301-393-0/krb5kdc/test-user.keytab.
Sep 25 04:48:50 dist-test-slave-kghz krb5kdc[22532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:48:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:48:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260925 04:48:50.133850 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790311449873301-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790311449873301-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790311449873301-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:41609
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:38579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:41609
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:48:50.248093 22545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:48:50.248361 22545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:48:50.248404 22545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:48:50.252480 22545 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:48:50.252545 22545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:48:50.252565 22545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:48:50.252583 22545 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:48:50.252597 22545 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:48:50.257920 22545 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:38579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790311449873301-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790311449873301-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:41609
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:41609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22545
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790311449873301-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:48:50.258922 22545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:48:50.259965 22545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:48:50.268329 22553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:48:50.268332 22550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:48:50.268613 22545 server_base.cc:1034] running on GCE node
W20260925 04:48:50.268337 22551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:48:50.269166 22545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:48:50.270344 22545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:48:50.271517 22545 hybrid_clock.cc:648] HybridClock initialized: now 1790311730271489 us; error 39 us; skew 500 ppm
Sep 25 04:48:50 dist-test-slave-kghz krb5kdc[22532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311730, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:48:50.274369 22545 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260925 04:48:50.275267 22545 webserver.cc:460] Webserver started at http://127.0.98.126:41177/ using document root <none> and password file <none>
I20260925 04:48:50.275839 22545 fs_manager.cc:359] Metadata directory not provided
I20260925 04:48:50.275907 22545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:48:50.276175 22545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:48:50.278319 22545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790311449873301-393-0/minicluster-data/master-0/data/instance:
uuid: "6ee10e81eb0b40c498cf27b6e5ab2f51"
format_stamp: "Formatted at 2026-09-25 04:48:50 on dist-test-slave-kghz"
server_key: "6846ed0d9cbcf3a97671e92fb7d64aa2"
server_key_iv: "9a2895fa16353ab809c38915ee78b2d5"
server_key_version: "encryptionkey@0"
I20260925 04:48:50.278939 22545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790311449873301-393-0/minicluster-data/master-0/wal/instance:
uuid: "6ee10e81eb0b40c498cf27b6e5ab2f51"
format_stamp: "Formatted at 2026-09-25 04:48:50 on dist-test-slave-kghz"
server_key: "6846ed0d9cbcf3a97671e92fb7d64aa2"
server_key_iv: "9a2895fa16353ab809c38915ee78b2d5"
server_key_version: "encryptionkey@0"
I20260925 04:48:50.282821 22545 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 04:48:50.285355 22560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:48:50.286828 22545 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:48:50.286983 22545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790311449873301-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790311449873301-393-0/minicluster-data/master-0/wal
uuid: "6ee10e81eb0b40c498cf27b6e5ab2f51"
format_stamp: "Formatted at 2026-09-25 04:48:50 on dist-test-slave-kghz"
server_key: "6846ed0d9cbcf3a97671e92fb7d64aa2"
server_key_iv: "9a2895fa16353ab809c38915ee78b2d5"
server_key_version: "encryptionkey@0"
I20260925 04:48:50.287109 22545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790311449873301-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790311449873301-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790311449873301-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:48:50.291761 22545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:48:50.294958 22545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:48:50.295158 22545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:48:50.301524 22545 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:41609
I20260925 04:48:50.301580 22612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:41609 every 8 connection(s)
I20260925 04:48:50.302560 22545 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
I20260925 04:48:50.305353 22613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:48:50.310526 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 22545
I20260925 04:48:50.310660 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790311449873301-393-0/minicluster-data/master-0/wal/instance
I20260925 04:48:50.310948 393 external_mini_cluster.cc:1428] Setting key 426cc727b696d9835c5bc3059dfc6088
I20260925 04:48:50.311085 22613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ee10e81eb0b40c498cf27b6e5ab2f51: Bootstrap starting.
I20260925 04:48:50.313618 22613 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ee10e81eb0b40c498cf27b6e5ab2f51: Neither blocks nor log segments found. Creating new log.
I20260925 04:48:50.314513 22613 log.cc:824] T 00000000000000000000000000000000 P 6ee10e81eb0b40c498cf27b6e5ab2f51: Log is configured to *not* fsync() on all Append() calls
I20260925 04:48:50.316511 22616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:50.312274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39223 (local address 127.0.98.126:41609)
0925 04:48:50.312725 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:50.312739 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:50.312776 (+ 37us) server_negotiation.cc:407] Connection header received
0925 04:48:50.313420 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:50.313437 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:50.313770 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:50.314095 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:50.314921 (+ 826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:50.314926 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:50.314938 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:48:50.314961 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:50.315016 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:48:50.315036 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:50.315123 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:50.315286 (+ 163us) server_negotiation.cc:299] Negotiation successful
0925 04:48:50.315607 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":153,"threads_started":1}
I20260925 04:48:50.317539 22613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ee10e81eb0b40c498cf27b6e5ab2f51: No bootstrap required, opened a new log
I20260925 04:48:50.319281 393 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:48:50.321476 22613 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ee10e81eb0b40c498cf27b6e5ab2f51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ee10e81eb0b40c498cf27b6e5ab2f51" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41609 } }
I20260925 04:48:50.321782 22613 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ee10e81eb0b40c498cf27b6e5ab2f51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:48:50.321913 22613 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ee10e81eb0b40c498cf27b6e5ab2f51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ee10e81eb0b40c498cf27b6e5ab2f51, State: Initialized, Role: FOLLOWER
I20260925 04:48:50.322480 22613 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ee10e81eb0b40c498cf27b6e5ab2f51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ee10e81eb0b40c498cf27b6e5ab2f51" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41609 } }
I20260925 04:48:50.322657 22613 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ee10e81eb0b40c498cf27b6e5ab2f51 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:48:50.322726 22613 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ee10e81eb0b40c498cf27b6e5ab2f51 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:48:50.322824 22613 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ee10e81eb0b40c498cf27b6e5ab2f51 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:48:50.323083 22616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:50.321325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55713 (local address 127.0.98.126:41609)
0925 04:48:50.321454 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:50.321464 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:50.321599 (+ 135us) server_negotiation.cc:407] Connection header received
0925 04:48:50.321727 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:50.321729 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:50.321777 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:50.321875 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:50.322629 (+ 754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:50.322632 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:50.322635 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:48:50.322647 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:50.322671 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:48:50.322686 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:50.322761 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:50.322925 (+ 164us) server_negotiation.cc:299] Negotiation successful
0925 04:48:50.322963 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 04:48:50.323884 22613 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ee10e81eb0b40c498cf27b6e5ab2f51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ee10e81eb0b40c498cf27b6e5ab2f51" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41609 } }
I20260925 04:48:50.324229 22613 leader_election.cc:302] T 00000000000000000000000000000000 P 6ee10e81eb0b40c498cf27b6e5ab2f51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ee10e81eb0b40c498cf27b6e5ab2f51; no voters:
I20260925 04:48:50.324502 22613 leader_election.cc:288] T 00000000000000000000000000000000 P 6ee10e81eb0b40c498cf27b6e5ab2f51 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:48:50.324626 22623 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ee10e81eb0b40c498cf27b6e5ab2f51 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:48:50.324862 22623 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ee10e81eb0b40c498cf27b6e5ab2f51 [term 1 LEADER]: Becoming Leader. State: Replica: 6ee10e81eb0b40c498cf27b6e5ab2f51, State: Running, Role: LEADER
I20260925 04:48:50.325205 22623 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ee10e81eb0b40c498cf27b6e5ab2f51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ee10e81eb0b40c498cf27b6e5ab2f51" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41609 } }
I20260925 04:48:50.325759 22613 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ee10e81eb0b40c498cf27b6e5ab2f51 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:48:50.326989 22624 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ee10e81eb0b40c498cf27b6e5ab2f51 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6ee10e81eb0b40c498cf27b6e5ab2f51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ee10e81eb0b40c498cf27b6e5ab2f51" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41609 } } }
I20260925 04:48:50.327121 22624 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ee10e81eb0b40c498cf27b6e5ab2f51 [sys.catalog]: This master's current role is: LEADER
I20260925 04:48:50.327492 22632 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:48:50.327461 22625 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ee10e81eb0b40c498cf27b6e5ab2f51 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ee10e81eb0b40c498cf27b6e5ab2f51. Latest consensus state: current_term: 1 leader_uuid: "6ee10e81eb0b40c498cf27b6e5ab2f51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ee10e81eb0b40c498cf27b6e5ab2f51" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41609 } } }
I20260925 04:48:50.329612 22625 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ee10e81eb0b40c498cf27b6e5ab2f51 [sys.catalog]: This master's current role is: LEADER
I20260925 04:48:50.330701 22632 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:48:50.336066 22632 catalog_manager.cc:1347] Generated new cluster ID: ab4f6078059a46fdaf73e30c92d4d744
I20260925 04:48:50.336130 22632 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:48:50.343537 22632 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:48:50.343840 22632 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:48:50.350849 22632 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ee10e81eb0b40c498cf27b6e5ab2f51: Generated new TSK 0
I20260925 04:48:50.351514 22632 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:48:50.365346 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 22545
2026-09-25T04:48:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1396 ms)
[----------] 18 tests from AuthTokenIssuingTest (25673 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311449873301-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:48:50 dist-test-slave-kghz krb5kdc[22644](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:48:50 dist-test-slave-kghz krb5kdc[22644](info): set up 2 sockets
Sep 25 04:48:50 dist-test-slave-kghz krb5kdc[22644](info): commencing operation
krb5kdc: starting...
W20260925 04:48:51.402647 393 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311449873301-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311449873301-393-0/krb5kdc/test-user.keytab.
Sep 25 04:48:51 dist-test-slave-kghz krb5kdc[22644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311731, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:48:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:48:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260925 04:48:51.529767 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311449873301-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311449873301-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311449873301-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:39091
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:36637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:39091,127.0.98.125:40911,127.0.98.124:43803
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:48:51.643851 22657 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 04:48:51.644176 22657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:48:51.644202 22657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:48:51.644289 22657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:48:51.648473 22657 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:48:51.648537 22657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:48:51.648557 22657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:48:51.648574 22657 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:48:51.648617 22657 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:48:51.653908 22657 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:36637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311449873301-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311449873301-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:39091,127.0.98.125:40911,127.0.98.124:43803
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:39091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22657
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311449873301-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:48:51.655107 22657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:48:51.656189 22657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:48:51.664976 22662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:48:51.665141 22657 server_base.cc:1034] running on GCE node
W20260925 04:48:51.665000 22665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:48:51.664999 22663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:48:51.665943 22657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:48:51.667201 22657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:48:51.668394 22657 hybrid_clock.cc:648] HybridClock initialized: now 1790311731668376 us; error 37 us; skew 500 ppm
Sep 25 04:48:51 dist-test-slave-kghz krb5kdc[22644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311731, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:48:51.672006 22657 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260925 04:48:51.673237 22657 webserver.cc:460] Webserver started at http://127.0.98.126:40445/ using document root <none> and password file <none>
I20260925 04:48:51.673830 22657 fs_manager.cc:359] Metadata directory not provided
I20260925 04:48:51.673929 22657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:48:51.674170 22657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:48:51.676311 22657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311449873301-393-0/minicluster-data/master-0/data/instance:
uuid: "920b12ae02fb43baa3f58ada6741ac04"
format_stamp: "Formatted at 2026-09-25 04:48:51 on dist-test-slave-kghz"
server_key: "1f5b843a38cf3445a25c803e34b3a8e4"
server_key_iv: "3be4cc3e00a6c921cf6e1c004e29458f"
server_key_version: "encryptionkey@0"
I20260925 04:48:51.676910 22657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311449873301-393-0/minicluster-data/master-0/wal/instance:
uuid: "920b12ae02fb43baa3f58ada6741ac04"
format_stamp: "Formatted at 2026-09-25 04:48:51 on dist-test-slave-kghz"
server_key: "1f5b843a38cf3445a25c803e34b3a8e4"
server_key_iv: "3be4cc3e00a6c921cf6e1c004e29458f"
server_key_version: "encryptionkey@0"
I20260925 04:48:51.680779 22657 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260925 04:48:51.683442 22672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:48:51.684794 22657 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:48:51.684937 22657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311449873301-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311449873301-393-0/minicluster-data/master-0/wal
uuid: "920b12ae02fb43baa3f58ada6741ac04"
format_stamp: "Formatted at 2026-09-25 04:48:51 on dist-test-slave-kghz"
server_key: "1f5b843a38cf3445a25c803e34b3a8e4"
server_key_iv: "3be4cc3e00a6c921cf6e1c004e29458f"
server_key_version: "encryptionkey@0"
I20260925 04:48:51.685057 22657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311449873301-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311449873301-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311449873301-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:48:51.705534 22657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:48:51.708752 22657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:48:51.708994 22657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:48:51.716687 22657 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:39091
I20260925 04:48:51.716997 22724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:39091 every 8 connection(s)
I20260925 04:48:51.717800 22657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
I20260925 04:48:51.720685 22725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:48:51.724395 22725 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39091 } has no permanent_uuid. Determining permanent_uuid...
I20260925 04:48:51.727309 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 22657
I20260925 04:48:51.727464 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311449873301-393-0/minicluster-data/master-0/wal/instance
I20260925 04:48:51.727762 393 external_mini_cluster.cc:1428] Setting key 3571ae1012e51e6f8876aa141e9982ce
Sep 25 04:48:51 dist-test-slave-kghz krb5kdc[22644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311731, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.125@KRBTEST.COM" created.
I20260925 04:48:51.741842 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:51.726612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39091 (local address 127.0.0.1:37912)
0925 04:48:51.727088 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:51.727149 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0925 04:48:51.728277 (+ 1128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:48:51.728994 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:48:51.729003 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:48:51.729507 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:48:51.729797 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:51.729811 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:51.731668 (+ 1857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:51.731673 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:51.732890 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:51.732900 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:51.733894 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:51.733897 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:51.733998 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:51.734709 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:48:51.734726 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:48:51.736542 (+ 1816us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:48:51.738558 (+ 2016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:48:51.738566 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:48:51.738575 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:48:51.738805 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:48:51.739225 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:48:51.739227 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:48:51.739228 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:48:51.739334 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:48:51.740073 (+ 739us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:48:51.740080 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:48:51.740227 (+ 147us) client_negotiation.cc:769] Sending connection context
0925 04:48:51.740413 (+ 186us) client_negotiation.cc:240] Negotiation successful
0925 04:48:51.740814 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"mutex_wait_us":111,"thread_start_us":102,"threads_started":1}
I20260925 04:48:51.741842 22728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:51.727358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37912 (local address 127.0.98.126:39091)
0925 04:48:51.727631 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:51.727644 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:51.727686 (+ 42us) server_negotiation.cc:407] Connection header received
0925 04:48:51.728608 (+ 922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:51.728613 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:51.728787 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:51.728941 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:51.729956 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:51.731485 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:51.733032 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:51.733744 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:51.733941 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:51.736738 (+ 2797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:51.736751 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:51.736762 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:48:51.736784 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:51.738459 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:51.738901 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:51.738982 (+ 81us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:51.738987 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:51.739142 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:51.739414 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:51.739417 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:51.739418 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:51.739937 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:51.740106 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:51.740486 (+ 380us) server_negotiation.cc:299] Negotiation successful
0925 04:48:51.740759 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":178,"thread_start_us":85,"threads_started":1}
I20260925 04:48:51.744362 22725 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.125" port: 40911 } has no permanent_uuid. Determining permanent_uuid...
I20260925 04:48:51.745175 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:51.744905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:40911 (local address 127.0.0.1:56912)
0925 04:48:51.745034 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:51.745112 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.125:40911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260925 04:48:51.745448 22676 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.125:40911: connect: Connection refused (error 111)
I20260925 04:48:51.746352 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:51.746101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:40911 (local address 127.0.0.1:56916)
0925 04:48:51.746208 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:51.746292 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.125:40911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 04:48:51.746582 22725 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.125:40911: Network error: Client connection negotiation failed: client connection to 127.0.98.125:40911: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311449873301-393-0/krb5kdc/kudu_127.0.98.125.keytab.
Entry for principal kudu/127.0.98.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311449873301-393-0/krb5kdc/kudu_127.0.98.125.keytab.
I20260925 04:48:51.758523 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311449873301-393-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311449873301-393-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311449873301-393-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311449873301-393-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.125:40911
--webserver_interface=127.0.98.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311449873301-393-0/krb5kdc/kudu_127.0.98.125.keytab
--principal=kudu/127.0.98.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:36637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:39091,127.0.98.125:40911,127.0.98.124:43803
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 04:48:51.795845 22725 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.125" port: 40911 } attempt: 1
I20260925 04:48:51.796803 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:51.796494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:40911 (local address 127.0.0.1:56928)
0925 04:48:51.796659 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:51.796724 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.125:40911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260925 04:48:51.797585 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:51.797353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:40911 (local address 127.0.0.1:56934)
0925 04:48:51.797488 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:51.797531 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.125:40911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 04:48:51.797714 22725 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.125:40911: Network error: Client connection negotiation failed: client connection to 127.0.98.125:40911: connect: Connection refused (error 111)
I20260925 04:48:51.865893 22725 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.125" port: 40911 } attempt: 2
I20260925 04:48:51.866788 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:51.866494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:40911 (local address 127.0.0.1:56946)
0925 04:48:51.866644 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:51.866710 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.125:40911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260925 04:48:51.867627 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:51.867387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:40911 (local address 127.0.0.1:56958)
0925 04:48:51.867509 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:51.867561 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.125:40911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 04:48:51.867779 22725 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.125:40911: Network error: Client connection negotiation failed: client connection to 127.0.98.125:40911: connect: Connection refused (error 111)
W20260925 04:48:51.875175 22732 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 04:48:51.875583 22732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:48:51.875679 22732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:48:51.875779 22732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:48:51.880752 22732 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:48:51.880841 22732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:48:51.880878 22732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:48:51.880920 22732 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:48:51.880955 22732 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:48:51.886536 22732 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:36637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311449873301-393-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311449873301-393-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:39091,127.0.98.125:40911,127.0.98.124:43803
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311449873301-393-0/krb5kdc/kudu_127.0.98.125.keytab
--principal=kudu/127.0.98.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.125:40911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311449873301-393-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22732
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311449873301-393-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:48:51.887652 22732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:48:51.888713 22732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 04:48:51.897572 22732 server_base.cc:1034] running on GCE node
W20260925 04:48:51.897615 22740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:48:51.897581 22737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:48:51.897581 22738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:48:51.898514 22732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:48:51.899632 22732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:48:51.900826 22732 hybrid_clock.cc:648] HybridClock initialized: now 1790311731900809 us; error 31 us; skew 500 ppm
Sep 25 04:48:51 dist-test-slave-kghz krb5kdc[22644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311731, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:48:51.909416 22732 init.cc:377] Logged in from keytab as kudu/127.0.98.125@KRBTEST.COM (short username kudu)
I20260925 04:48:51.910468 22732 webserver.cc:460] Webserver started at http://127.0.98.125:38019/ using document root <none> and password file <none>
I20260925 04:48:51.911006 22732 fs_manager.cc:359] Metadata directory not provided
I20260925 04:48:51.911065 22732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:48:51.911224 22732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:48:51.913223 22732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311449873301-393-0/minicluster-data/master-1/data/instance:
uuid: "428f0d0f5a9c4cc49873d4add804849d"
format_stamp: "Formatted at 2026-09-25 04:48:51 on dist-test-slave-kghz"
server_key: "1f56d735e98ead87085c72e442d7d3b5"
server_key_iv: "cd0a0bf4f1fc37c92ac28c68ad1de930"
server_key_version: "encryptionkey@0"
I20260925 04:48:51.913784 22732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311449873301-393-0/minicluster-data/master-1/wal/instance:
uuid: "428f0d0f5a9c4cc49873d4add804849d"
format_stamp: "Formatted at 2026-09-25 04:48:51 on dist-test-slave-kghz"
server_key: "1f56d735e98ead87085c72e442d7d3b5"
server_key_iv: "cd0a0bf4f1fc37c92ac28c68ad1de930"
server_key_version: "encryptionkey@0"
I20260925 04:48:51.917666 22732 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 04:48:51.920177 22747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:48:51.921536 22732 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 04:48:51.921690 22732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311449873301-393-0/minicluster-data/master-1/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311449873301-393-0/minicluster-data/master-1/wal
uuid: "428f0d0f5a9c4cc49873d4add804849d"
format_stamp: "Formatted at 2026-09-25 04:48:51 on dist-test-slave-kghz"
server_key: "1f56d735e98ead87085c72e442d7d3b5"
server_key_iv: "cd0a0bf4f1fc37c92ac28c68ad1de930"
server_key_version: "encryptionkey@0"
I20260925 04:48:51.921797 22732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311449873301-393-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311449873301-393-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311449873301-393-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:48:51.958983 22725 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.125" port: 40911 } attempt: 3
I20260925 04:48:51.959861 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:51.959509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:40911 (local address 127.0.0.1:56966)
0925 04:48:51.959689 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:51.959770 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.125:40911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260925 04:48:51.960624 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:51.960393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:40911 (local address 127.0.0.1:56978)
0925 04:48:51.960507 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:51.960565 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.125:40911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 04:48:51.960769 22725 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.125:40911: Network error: Client connection negotiation failed: client connection to 127.0.98.125:40911: connect: Connection refused (error 111)
I20260925 04:48:51.961526 22732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:48:51.964936 22732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:48:51.965150 22732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:48:51.972229 22732 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.125:40911
I20260925 04:48:51.972506 22799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.125:40911 every 8 connection(s)
I20260925 04:48:51.973268 22732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311449873301-393-0/minicluster-data/master-1/data/info.pb
I20260925 04:48:51.975448 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 22732
I20260925 04:48:51.975584 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311449873301-393-0/minicluster-data/master-1/wal/instance
I20260925 04:48:51.975894 393 external_mini_cluster.cc:1428] Setting key 357cfd1fc3a487ad227658ce68fdf99f
I20260925 04:48:51.976688 22800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:48:51.980861 22800 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39091 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.124@KRBTEST.COM" created.
Sep 25 04:48:51 dist-test-slave-kghz krb5kdc[22644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311731, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.125@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:48:51.996711 22728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:51.982940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37914 (local address 127.0.98.126:39091)
0925 04:48:51.983082 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:51.983086 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:51.984238 (+ 1152us) server_negotiation.cc:407] Connection header received
0925 04:48:51.985342 (+ 1104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:51.985346 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:51.985410 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:51.985527 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:51.986993 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:51.987584 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:51.989268 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:51.989695 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:51.989798 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:51.992861 (+ 3063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:51.992865 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:51.992869 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:48:51.992900 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:51.994490 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:51.995046 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:51.995055 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:51.995056 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:51.995131 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:51.995484 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:51.995489 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:51.995490 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:51.995681 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:51.995798 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:51.996401 (+ 603us) server_negotiation.cc:299] Negotiation successful
0925 04:48:51.996532 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 04:48:51.997531 22803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:51.983452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39091 (local address 127.0.0.1:37914)
0925 04:48:51.984069 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:51.984118 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 04:48:51.985114 (+ 996us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:48:51.985699 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:48:51.985706 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:48:51.986162 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:48:51.986785 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:51.986805 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:51.987716 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:51.987719 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:51.989079 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:51.989092 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:51.989883 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:51.989890 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:51.990131 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:51.990823 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:48:51.990850 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:48:51.992664 (+ 1814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:48:51.994624 (+ 1960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:48:51.994631 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:48:51.994648 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:48:51.994929 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:48:51.995227 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:48:51.995231 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:48:51.995233 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:48:51.995385 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:48:51.995789 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:48:51.995796 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:48:51.996134 (+ 338us) client_negotiation.cc:769] Sending connection context
0925 04:48:51.996337 (+ 203us) client_negotiation.cc:240] Negotiation successful
0925 04:48:51.996673 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":161,"threads_started":1}
I20260925 04:48:51.999112 22800 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.125" port: 40911 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311449873301-393-0/krb5kdc/kudu_127.0.98.124.keytab.
Entry for principal kudu/127.0.98.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311449873301-393-0/krb5kdc/kudu_127.0.98.124.keytab.
Sep 25 04:48:52 dist-test-slave-kghz krb5kdc[22644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311731, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.125@KRBTEST.COM for kudu/127.0.98.125@KRBTEST.COM
I20260925 04:48:52.005791 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311449873301-393-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311449873301-393-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311449873301-393-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311449873301-393-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.124:43803
--webserver_interface=127.0.98.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311449873301-393-0/krb5kdc/kudu_127.0.98.124.keytab
--principal=kudu/127.0.98.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:36637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:39091,127.0.98.125:40911,127.0.98.124:43803
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 04:48:52.009179 22803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:51.999730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:40911 (local address 127.0.0.1:56984)
0925 04:48:51.999907 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:51.999923 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:48:52.000010 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:48:52.000641 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:48:52.000644 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:48:52.000780 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:48:52.000919 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:52.000924 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:52.002092 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:52.002094 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:52.002789 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:52.002797 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:52.003580 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:52.003584 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:52.003671 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:52.004187 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:48:52.004198 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:48:52.005565 (+ 1367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:48:52.007626 (+ 2061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:48:52.007629 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:48:52.007631 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:48:52.007837 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:48:52.008124 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:48:52.008129 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:48:52.008131 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:48:52.008207 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:48:52.008661 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:48:52.008664 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:48:52.008745 (+ 81us) client_negotiation.cc:769] Sending connection context
0925 04:48:52.008860 (+ 115us) client_negotiation.cc:240] Negotiation successful
0925 04:48:52.008992 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97}
I20260925 04:48:52.009224 22805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:51.999888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56984 (local address 127.0.98.125:40911)
0925 04:48:52.000214 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:52.000223 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:52.000242 (+ 19us) server_negotiation.cc:407] Connection header received
0925 04:48:52.000277 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:52.000281 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:52.000445 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:52.000613 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:52.001022 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:52.001981 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:52.002931 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:52.003458 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:52.003546 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:52.005696 (+ 2150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:52.005705 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:52.005719 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:48:52.005746 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:52.007496 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:52.007955 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:52.007960 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:52.007962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:52.008037 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:52.008312 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:52.008315 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:52.008317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:52.008555 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:52.008689 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:52.008950 (+ 261us) server_negotiation.cc:299] Negotiation successful
0925 04:48:52.009074 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":117,"threads_started":1}
I20260925 04:48:52.011744 22800 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.124" port: 43803 } has no permanent_uuid. Determining permanent_uuid...
I20260925 04:48:52.012492 22803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:52.012220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:43803 (local address 127.0.0.1:37976)
0925 04:48:52.012355 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:52.012428 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:43803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 04:48:52.012686 22750 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.124:43803: connect: Connection refused (error 111)
I20260925 04:48:52.013480 22803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:52.013288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:43803 (local address 127.0.0.1:37990)
0925 04:48:52.013385 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:52.013432 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:43803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260925 04:48:52.013638 22800 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.124:43803: Network error: Client connection negotiation failed: client connection to 127.0.98.124:43803: connect: Connection refused (error 111)
I20260925 04:48:52.062878 22800 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.124" port: 43803 } attempt: 1
I20260925 04:48:52.063831 22803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:52.063480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:43803 (local address 127.0.0.1:38006)
0925 04:48:52.063643 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:52.063710 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:43803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260925 04:48:52.064574 22803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:52.064350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:43803 (local address 127.0.0.1:38020)
0925 04:48:52.064460 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:52.064513 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:43803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260925 04:48:52.064711 22800 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.124:43803: Network error: Client connection negotiation failed: client connection to 127.0.98.124:43803: connect: Connection refused (error 111)
I20260925 04:48:52.103929 22725 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.125" port: 40911 } attempt: 4
Sep 25 04:48:52 dist-test-slave-kghz krb5kdc[22644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311731, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for kudu/127.0.98.125@KRBTEST.COM
I20260925 04:48:52.113298 22805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:52.104651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56988 (local address 127.0.98.125:40911)
0925 04:48:52.104782 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:52.104787 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:52.104873 (+ 86us) server_negotiation.cc:407] Connection header received
0925 04:48:52.105070 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:52.105074 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:52.105124 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:52.105211 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:52.105656 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:52.106180 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:52.106981 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:52.107688 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:52.107811 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:52.110086 (+ 2275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:52.110090 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:52.110092 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:48:52.110115 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:52.111651 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:52.112089 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:52.112094 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:52.112096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:52.112176 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:52.112465 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:52.112468 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:52.112470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:52.112650 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:52.112756 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:52.113029 (+ 273us) server_negotiation.cc:299] Negotiation successful
0925 04:48:52.113142 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 04:48:52.113298 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:52.104555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:40911 (local address 127.0.0.1:56988)
0925 04:48:52.104764 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:52.104781 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 04:48:52.104934 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:48:52.105231 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:48:52.105234 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:48:52.105433 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:48:52.105527 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:52.105533 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:52.106316 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:52.106321 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:52.106850 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:52.106858 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:52.107815 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:52.107819 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:52.107934 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:52.108464 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:48:52.108477 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:48:52.109913 (+ 1436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:48:52.111777 (+ 1864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:48:52.111781 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:48:52.111783 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:48:52.111991 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:48:52.112268 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:48:52.112272 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:48:52.112275 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:48:52.112362 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:48:52.112746 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:48:52.112749 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:48:52.112855 (+ 106us) client_negotiation.cc:769] Sending connection context
0925 04:48:52.112975 (+ 120us) client_negotiation.cc:240] Negotiation successful
0925 04:48:52.113124 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111}
I20260925 04:48:52.114513 22725 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.124" port: 43803 } has no permanent_uuid. Determining permanent_uuid...
I20260925 04:48:52.115195 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:52.114959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:43803 (local address 127.0.0.1:38034)
0925 04:48:52.115071 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:52.115124 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:43803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 04:48:52.115852 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:52.115644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:43803 (local address 127.0.0.1:38050)
0925 04:48:52.115750 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:52.115796 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:43803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 04:48:52.116000 22725 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.124:43803: Network error: Client connection negotiation failed: client connection to 127.0.98.124:43803: connect: Connection refused (error 111)
W20260925 04:48:52.122148 22806 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 04:48:52.122566 22806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:48:52.122629 22806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:48:52.122709 22806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:48:52.126967 22806 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:48:52.127051 22806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:48:52.127086 22806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:48:52.127121 22806 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:48:52.127154 22806 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:48:52.132710 22806 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:36637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311449873301-393-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311449873301-393-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:39091,127.0.98.125:40911,127.0.98.124:43803
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311449873301-393-0/krb5kdc/kudu_127.0.98.124.keytab
--principal=kudu/127.0.98.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.124:43803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311449873301-393-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22806
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311449873301-393-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:48:52.132884 22800 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.124" port: 43803 } attempt: 2
I20260925 04:48:52.133699 22803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:52.133399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:43803 (local address 127.0.0.1:38056)
0925 04:48:52.133555 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:52.133624 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:43803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260925 04:48:52.133935 22806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:48:52.134517 22803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:52.134298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:43803 (local address 127.0.0.1:38072)
0925 04:48:52.134411 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:52.134462 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:43803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 04:48:52.134644 22800 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.124:43803: Network error: Client connection negotiation failed: client connection to 127.0.98.124:43803: connect: Connection refused (error 111)
I20260925 04:48:52.135214 22806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:48:52.144156 22813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:48:52.144191 22815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:48:52.144203 22812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:48:52.144549 22806 server_base.cc:1034] running on GCE node
I20260925 04:48:52.145169 22806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:48:52.146402 22806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:48:52.147568 22806 hybrid_clock.cc:648] HybridClock initialized: now 1790311732147552 us; error 29 us; skew 500 ppm
Sep 25 04:48:52 dist-test-slave-kghz krb5kdc[22644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311732, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:48:52.150375 22806 init.cc:377] Logged in from keytab as kudu/127.0.98.124@KRBTEST.COM (short username kudu)
I20260925 04:48:52.151268 22806 webserver.cc:460] Webserver started at http://127.0.98.124:40433/ using document root <none> and password file <none>
I20260925 04:48:52.151779 22806 fs_manager.cc:359] Metadata directory not provided
I20260925 04:48:52.151836 22806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:48:52.152007 22806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:48:52.154052 22806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311449873301-393-0/minicluster-data/master-2/data/instance:
uuid: "7397548a92234639b270117b2aa46274"
format_stamp: "Formatted at 2026-09-25 04:48:52 on dist-test-slave-kghz"
server_key: "ec36654752449eafd3de35ee2c108cda"
server_key_iv: "7859683f6005f7aa6db9088711ac0bab"
server_key_version: "encryptionkey@0"
I20260925 04:48:52.154626 22806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311449873301-393-0/minicluster-data/master-2/wal/instance:
uuid: "7397548a92234639b270117b2aa46274"
format_stamp: "Formatted at 2026-09-25 04:48:52 on dist-test-slave-kghz"
server_key: "ec36654752449eafd3de35ee2c108cda"
server_key_iv: "7859683f6005f7aa6db9088711ac0bab"
server_key_version: "encryptionkey@0"
I20260925 04:48:52.158237 22806 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:48:52.160604 22822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:48:52.161795 22806 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 04:48:52.161971 22806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311449873301-393-0/minicluster-data/master-2/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311449873301-393-0/minicluster-data/master-2/wal
uuid: "7397548a92234639b270117b2aa46274"
format_stamp: "Formatted at 2026-09-25 04:48:52 on dist-test-slave-kghz"
server_key: "ec36654752449eafd3de35ee2c108cda"
server_key_iv: "7859683f6005f7aa6db9088711ac0bab"
server_key_version: "encryptionkey@0"
I20260925 04:48:52.162088 22806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311449873301-393-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311449873301-393-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311449873301-393-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:48:52.175143 22725 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.124" port: 43803 } attempt: 1
I20260925 04:48:52.175973 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:52.175691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:43803 (local address 127.0.0.1:38076)
0925 04:48:52.175836 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:52.175898 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:43803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260925 04:48:52.176683 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:52.176465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:43803 (local address 127.0.0.1:38086)
0925 04:48:52.176575 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:52.176625 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:43803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260925 04:48:52.176812 22725 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.124:43803: Network error: Client connection negotiation failed: client connection to 127.0.98.124:43803: connect: Connection refused (error 111)
I20260925 04:48:52.179534 22806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:48:52.182730 22806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:48:52.182941 22806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:48:52.190716 22806 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.124:43803
I20260925 04:48:52.191022 22874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.124:43803 every 8 connection(s)
I20260925 04:48:52.191718 22806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311449873301-393-0/minicluster-data/master-2/data/info.pb
I20260925 04:48:52.193969 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 22806
I20260925 04:48:52.194119 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790311449873301-393-0/minicluster-data/master-2/wal/instance
I20260925 04:48:52.194445 393 external_mini_cluster.cc:1428] Setting key c61c4f6d786eb485f9f41fc4063aa6f0
I20260925 04:48:52.194623 22875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:48:52.199513 22875 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39091 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 04:48:52 dist-test-slave-kghz krb5kdc[22644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311731, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.124@KRBTEST.COM
Sep 25 04:48:52 dist-test-slave-kghz krb5kdc[22644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311732, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.124@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:48:52.210754 22878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:52.195858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38094 (local address 127.0.98.124:43803)
0925 04:48:52.196286 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:52.196301 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:52.196336 (+ 35us) server_negotiation.cc:407] Connection header received
0925 04:48:52.197221 (+ 885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:52.197239 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:52.197563 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:52.197859 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:52.198817 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:52.199821 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:52.201054 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:52.201752 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:52.201986 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:52.205044 (+ 3058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:52.205050 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:52.205059 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:48:52.205085 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:52.207035 (+ 1950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:52.208096 (+ 1061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:52.208103 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:52.208116 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:52.208245 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:52.208640 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:52.208643 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:52.208644 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:52.209048 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:52.209266 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:52.209553 (+ 287us) server_negotiation.cc:299] Negotiation successful
0925 04:48:52.209866 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":150,"threads_started":1}
I20260925 04:48:52.212488 22728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:52.201606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37926 (local address 127.0.98.126:39091)
0925 04:48:52.201776 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:52.201781 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:52.202111 (+ 330us) server_negotiation.cc:407] Connection header received
0925 04:48:52.202436 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:52.202440 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:52.202497 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:52.202585 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:52.203622 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:52.204350 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:52.205356 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:52.205684 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:52.205801 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:52.208220 (+ 2419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:52.208223 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:52.208225 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:48:52.208252 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:52.210655 (+ 2403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:52.211184 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:52.211187 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:52.211188 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:52.211263 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:52.211563 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:52.211567 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:52.211568 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:52.211749 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:52.211862 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:52.212214 (+ 352us) server_negotiation.cc:299] Negotiation successful
0925 04:48:52.212342 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260925 04:48:52.212500 22879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:52.201467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39091 (local address 127.0.0.1:37926)
0925 04:48:52.201956 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:52.202014 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 04:48:52.202225 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:48:52.202721 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:48:52.202727 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:48:52.203265 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:48:52.203471 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:52.203481 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:52.204522 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:52.204528 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:52.205222 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:52.205231 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:52.205818 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:52.205849 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:52.205936 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:52.206523 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:48:52.206535 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:48:52.208002 (+ 1467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:48:52.210768 (+ 2766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:48:52.210778 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:48:52.210784 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:48:52.211062 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:48:52.211370 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:48:52.211375 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:48:52.211377 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:48:52.211467 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:48:52.211864 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:48:52.211871 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:48:52.211984 (+ 113us) client_negotiation.cc:769] Sending connection context
0925 04:48:52.212129 (+ 145us) client_negotiation.cc:240] Negotiation successful
0925 04:48:52.212316 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":140,"threads_started":1}
I20260925 04:48:52.213500 22875 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.125" port: 40911 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 04:48:52 dist-test-slave-kghz krb5kdc[22644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311732, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.124@KRBTEST.COM for kudu/127.0.98.125@KRBTEST.COM
Sep 25 04:48:52 dist-test-slave-kghz krb5kdc[22644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311731, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:48:52.222209 22805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:52.214124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56994 (local address 127.0.98.125:40911)
0925 04:48:52.214284 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:52.214289 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:52.214305 (+ 16us) server_negotiation.cc:407] Connection header received
0925 04:48:52.214348 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:52.214351 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:52.214409 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:52.214500 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:52.214908 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:52.215512 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:52.216364 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:52.216702 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:52.216818 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:52.218775 (+ 1957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:52.218779 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:52.218781 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:48:52.218812 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:52.220512 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:52.220965 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:52.220968 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:52.220969 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:52.221037 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:52.221322 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:52.221328 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:52.221332 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:52.221547 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:52.221644 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:52.221964 (+ 320us) server_negotiation.cc:299] Negotiation successful
0925 04:48:52.222089 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 04:48:52.222213 22879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:52.214019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:40911 (local address 127.0.0.1:56994)
0925 04:48:52.214134 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:52.214149 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:48:52.214222 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:48:52.214527 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:48:52.214530 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:48:52.214692 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:48:52.214796 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:52.214802 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:52.215619 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:52.215621 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:52.216241 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:52.216251 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:52.216836 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:52.216841 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:52.216953 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:52.217399 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:48:52.217411 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:48:52.218639 (+ 1228us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:48:52.220616 (+ 1977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:48:52.220622 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:48:52.220626 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:48:52.220855 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:48:52.221127 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:48:52.221131 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:48:52.221133 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:48:52.221231 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:48:52.221678 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:48:52.221681 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:48:52.221782 (+ 101us) client_negotiation.cc:769] Sending connection context
0925 04:48:52.221894 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 04:48:52.222019 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 04:48:52.222976 22875 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.124" port: 43803 } has no permanent_uuid. Determining permanent_uuid...
I20260925 04:48:52.223202 22728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:52.213204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37930 (local address 127.0.98.126:39091)
0925 04:48:52.213300 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:52.213304 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:52.213360 (+ 56us) server_negotiation.cc:407] Connection header received
0925 04:48:52.213505 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:52.213508 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:52.213557 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:52.213644 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:52.214464 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:52.215181 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:52.216045 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:52.216373 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:52.216478 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:52.219422 (+ 2944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:52.219425 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:52.219428 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:48:52.219451 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:52.221169 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:52.221675 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:52.221679 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:52.221681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:52.221771 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:52.222095 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:52.222099 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:52.222100 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:52.222367 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:52.222603 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:52.222950 (+ 347us) server_negotiation.cc:299] Negotiation successful
0925 04:48:52.223070 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 04:48:52.225798 22800 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.124" port: 43803 } attempt: 3
Sep 25 04:48:52 dist-test-slave-kghz krb5kdc[22644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311732, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.124@KRBTEST.COM for kudu/127.0.98.124@KRBTEST.COM
Sep 25 04:48:52 dist-test-slave-kghz krb5kdc[22644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311731, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.125@KRBTEST.COM
Sep 25 04:48:52 dist-test-slave-kghz krb5kdc[22644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311731, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.125@KRBTEST.COM for kudu/127.0.98.124@KRBTEST.COM
I20260925 04:48:52.232440 22879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:52.223470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:43803 (local address 127.0.0.1:38104)
0925 04:48:52.223571 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:52.223585 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:48:52.223659 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:48:52.223856 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:48:52.223859 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:48:52.224025 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:48:52.224083 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:52.224088 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:52.225299 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:52.225302 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:52.226189 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:52.226199 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:52.226829 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:52.226832 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:52.226914 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:52.227391 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:48:52.227406 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:48:52.228787 (+ 1381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:48:52.230806 (+ 2019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:48:52.230809 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:48:52.230818 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:48:52.231054 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:48:52.231330 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:48:52.231336 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:48:52.231341 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:48:52.231415 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:48:52.231818 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:48:52.231821 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:48:52.231929 (+ 108us) client_negotiation.cc:769] Sending connection context
0925 04:48:52.232054 (+ 125us) client_negotiation.cc:240] Negotiation successful
0925 04:48:52.232189 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 04:48:52.232777 22878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:52.223523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38104 (local address 127.0.98.124:43803)
0925 04:48:52.223622 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:52.223626 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:52.223638 (+ 12us) server_negotiation.cc:407] Connection header received
0925 04:48:52.223729 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:52.223734 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:52.223766 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:52.223850 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:52.224191 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:52.225155 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:52.226321 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:52.226647 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:52.226737 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:52.228917 (+ 2180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:52.228920 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:52.228923 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:48:52.228954 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:52.230705 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:52.231155 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:52.231157 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:52.231160 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:52.231233 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:52.231506 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:52.231509 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:52.231510 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:52.231699 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:52.231807 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:52.232136 (+ 329us) server_negotiation.cc:299] Negotiation successful
0925 04:48:52.232247 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 04:48:52.234158 22805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:52.224340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57004 (local address 127.0.98.125:40911)
0925 04:48:52.224443 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:52.224446 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:52.224458 (+ 12us) server_negotiation.cc:407] Connection header received
0925 04:48:52.224541 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:52.224543 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:52.224573 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:52.224633 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:52.225342 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:52.226070 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:52.227479 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:52.227829 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:52.227947 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:52.230521 (+ 2574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:52.230524 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:52.230526 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:48:52.230546 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:52.232132 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:52.232618 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:52.232624 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:52.232627 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:52.232716 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:52.233017 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:52.233020 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:52.233022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:52.233261 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:52.233498 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:52.233859 (+ 361us) server_negotiation.cc:299] Negotiation successful
0925 04:48:52.233990 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 04:48:52.236577 22803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:52.226181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:43803 (local address 127.0.0.1:38108)
0925 04:48:52.226339 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:52.226359 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 04:48:52.226447 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:48:52.227067 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:48:52.227070 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:48:52.227232 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:48:52.227302 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:52.227306 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:52.228759 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:52.228764 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:52.229914 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:52.229929 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:52.230632 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:52.230639 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:52.230731 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:52.231208 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:48:52.231222 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:48:52.232680 (+ 1458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:48:52.234969 (+ 2289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:48:52.234972 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:48:52.234974 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:48:52.235219 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:48:52.235540 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:48:52.235543 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:48:52.235545 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:48:52.235638 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:48:52.236052 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:48:52.236056 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:48:52.236154 (+ 98us) client_negotiation.cc:769] Sending connection context
0925 04:48:52.236278 (+ 124us) client_negotiation.cc:240] Negotiation successful
0925 04:48:52.236409 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95}
I20260925 04:48:52.236598 22880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:52.226227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38108 (local address 127.0.98.124:43803)
0925 04:48:52.226867 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:52.226875 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:52.226888 (+ 13us) server_negotiation.cc:407] Connection header received
0925 04:48:52.226924 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:52.226926 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:52.226958 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:52.227053 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:52.227448 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:52.228649 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:52.230054 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:52.230486 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:52.230933 (+ 447us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:52.232824 (+ 1891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:52.232830 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:52.232834 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:48:52.232864 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:52.234856 (+ 1992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:52.235339 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:52.235342 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:52.235344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:52.235439 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:52.235744 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:52.235747 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:52.235749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:52.235945 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:52.236041 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:52.236330 (+ 289us) server_negotiation.cc:299] Negotiation successful
0925 04:48:52.236444 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":114,"threads_started":1}
I20260925 04:48:52.236712 22875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7397548a92234639b270117b2aa46274: Bootstrap starting.
I20260925 04:48:52.239650 22800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 428f0d0f5a9c4cc49873d4add804849d: Bootstrap starting.
I20260925 04:48:52.239745 22875 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7397548a92234639b270117b2aa46274: Neither blocks nor log segments found. Creating new log.
I20260925 04:48:52.239739 393 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:48:52.240624 22875 log.cc:824] T 00000000000000000000000000000000 P 7397548a92234639b270117b2aa46274: Log is configured to *not* fsync() on all Append() calls
I20260925 04:48:52.242544 22800 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 428f0d0f5a9c4cc49873d4add804849d: Neither blocks nor log segments found. Creating new log.
I20260925 04:48:52.241227 22699 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "920b12ae02fb43baa3f58ada6741ac04"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37930
I20260925 04:48:52.243494 22800 log.cc:824] T 00000000000000000000000000000000 P 428f0d0f5a9c4cc49873d4add804849d: Log is configured to *not* fsync() on all Append() calls
I20260925 04:48:52.243935 22725 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.124" port: 43803 } attempt: 2
I20260925 04:48:52.244629 22875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7397548a92234639b270117b2aa46274: No bootstrap required, opened a new log
I20260925 04:48:52.245918 22800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 428f0d0f5a9c4cc49873d4add804849d: No bootstrap required, opened a new log
I20260925 04:48:52.248474 22875 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7397548a92234639b270117b2aa46274 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920b12ae02fb43baa3f58ada6741ac04" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39091 } } peers { permanent_uuid: "428f0d0f5a9c4cc49873d4add804849d" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 40911 } } peers { permanent_uuid: "7397548a92234639b270117b2aa46274" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 43803 } }
I20260925 04:48:52.248665 22875 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7397548a92234639b270117b2aa46274 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7397548a92234639b270117b2aa46274, State: Initialized, Role: FOLLOWER
I20260925 04:48:52.248929 22875 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7397548a92234639b270117b2aa46274 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920b12ae02fb43baa3f58ada6741ac04" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39091 } } peers { permanent_uuid: "428f0d0f5a9c4cc49873d4add804849d" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 40911 } } peers { permanent_uuid: "7397548a92234639b270117b2aa46274" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 43803 } }
Sep 25 04:48:52 dist-test-slave-kghz krb5kdc[22644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311731, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for kudu/127.0.98.124@KRBTEST.COM
I20260925 04:48:52.249343 22883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7397548a92234639b270117b2aa46274 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920b12ae02fb43baa3f58ada6741ac04" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39091 } } peers { permanent_uuid: "428f0d0f5a9c4cc49873d4add804849d" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 40911 } } peers { permanent_uuid: "7397548a92234639b270117b2aa46274" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 43803 } } }
I20260925 04:48:52.249476 22883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7397548a92234639b270117b2aa46274 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:48:52.250171 22875 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7397548a92234639b270117b2aa46274 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:48:52.250064 22800 raft_consensus.cc:348] T 00000000000000000000000000000000 P 428f0d0f5a9c4cc49873d4add804849d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920b12ae02fb43baa3f58ada6741ac04" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39091 } } peers { permanent_uuid: "428f0d0f5a9c4cc49873d4add804849d" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 40911 } } peers { permanent_uuid: "7397548a92234639b270117b2aa46274" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 43803 } }
I20260925 04:48:52.250315 22800 raft_consensus.cc:729] T 00000000000000000000000000000000 P 428f0d0f5a9c4cc49873d4add804849d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 428f0d0f5a9c4cc49873d4add804849d, State: Initialized, Role: FOLLOWER
I20260925 04:48:52.250639 22800 consensus_queue.cc:260] T 00000000000000000000000000000000 P 428f0d0f5a9c4cc49873d4add804849d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920b12ae02fb43baa3f58ada6741ac04" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39091 } } peers { permanent_uuid: "428f0d0f5a9c4cc49873d4add804849d" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 40911 } } peers { permanent_uuid: "7397548a92234639b270117b2aa46274" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 43803 } }
I20260925 04:48:52.251184 22884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 428f0d0f5a9c4cc49873d4add804849d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920b12ae02fb43baa3f58ada6741ac04" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39091 } } peers { permanent_uuid: "428f0d0f5a9c4cc49873d4add804849d" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 40911 } } peers { permanent_uuid: "7397548a92234639b270117b2aa46274" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 43803 } } }
I20260925 04:48:52.251513 22884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 428f0d0f5a9c4cc49873d4add804849d [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:48:52.251979 22800 sys_catalog.cc:564] T 00000000000000000000000000000000 P 428f0d0f5a9c4cc49873d4add804849d [sys.catalog]: configured and running, proceeding with master startup.
W20260925 04:48:52.256525 22895 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7397548a92234639b270117b2aa46274: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 04:48:52.256673 22895 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 04:48:52.254451 22806 master_runner.cc:186] Error getting master registration for 127.0.98.126:39091: OK, instance_id { permanent_uuid: "920b12ae02fb43baa3f58ada6741ac04" instance_seqno: 1790311731714612 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 04:48:52.257805 22806 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 04:48:52.255239 22732 master_runner.cc:186] Error getting master registration for 127.0.98.126:39091: OK, instance_id { permanent_uuid: "920b12ae02fb43baa3f58ada6741ac04" instance_seqno: 1790311731714612 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 04:48:52.258766 22732 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 04:48:52.258876 22880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:52.244442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38124 (local address 127.0.98.124:43803)
0925 04:48:52.244552 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:52.244556 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:52.244572 (+ 16us) server_negotiation.cc:407] Connection header received
0925 04:48:52.244675 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:52.244678 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:52.244726 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:52.244815 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:52.245273 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:52.246011 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:52.247063 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:52.247423 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:52.247536 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:52.250100 (+ 2564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:52.250103 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:52.250106 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:48:52.250129 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:52.252636 (+ 2507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:52.253204 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:52.253207 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:52.253209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:52.253304 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:52.257909 (+ 4605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:52.257914 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:52.257916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:52.258165 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:52.258308 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:52.258617 (+ 309us) server_negotiation.cc:299] Negotiation successful
0925 04:48:52.258740 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 04:48:52.258877 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:52.244373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:43803 (local address 127.0.0.1:38124)
0925 04:48:52.244484 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:52.244502 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 04:48:52.244587 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:48:52.244841 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:48:52.244844 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:48:52.245036 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:48:52.245141 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:52.245148 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:52.246127 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:52.246131 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:52.246935 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:52.246946 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:52.247544 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:52.247548 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:52.247637 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:52.248173 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:48:52.248187 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:48:52.249958 (+ 1771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:48:52.252774 (+ 2816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:48:52.252779 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:48:52.252784 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:48:52.253076 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:48:52.253431 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:48:52.253436 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:48:52.253438 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:48:52.253534 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:48:52.258302 (+ 4768us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:48:52.258305 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:48:52.258423 (+ 118us) client_negotiation.cc:769] Sending connection context
0925 04:48:52.258549 (+ 126us) client_negotiation.cc:240] Negotiation successful
0925 04:48:52.258714 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
W20260925 04:48:52.259215 22906 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 428f0d0f5a9c4cc49873d4add804849d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 04:48:52.259299 22906 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 04:48:52.261528 22725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 920b12ae02fb43baa3f58ada6741ac04: Bootstrap starting.
I20260925 04:48:52.264221 22725 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 920b12ae02fb43baa3f58ada6741ac04: Neither blocks nor log segments found. Creating new log.
I20260925 04:48:52.265130 22725 log.cc:824] T 00000000000000000000000000000000 P 920b12ae02fb43baa3f58ada6741ac04: Log is configured to *not* fsync() on all Append() calls
I20260925 04:48:52.267521 22725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 920b12ae02fb43baa3f58ada6741ac04: No bootstrap required, opened a new log
I20260925 04:48:52.270651 22725 raft_consensus.cc:348] T 00000000000000000000000000000000 P 920b12ae02fb43baa3f58ada6741ac04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920b12ae02fb43baa3f58ada6741ac04" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39091 } } peers { permanent_uuid: "428f0d0f5a9c4cc49873d4add804849d" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 40911 } } peers { permanent_uuid: "7397548a92234639b270117b2aa46274" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 43803 } }
I20260925 04:48:52.270987 22725 raft_consensus.cc:729] T 00000000000000000000000000000000 P 920b12ae02fb43baa3f58ada6741ac04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 920b12ae02fb43baa3f58ada6741ac04, State: Initialized, Role: FOLLOWER
I20260925 04:48:52.271410 22725 consensus_queue.cc:260] T 00000000000000000000000000000000 P 920b12ae02fb43baa3f58ada6741ac04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920b12ae02fb43baa3f58ada6741ac04" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39091 } } peers { permanent_uuid: "428f0d0f5a9c4cc49873d4add804849d" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 40911 } } peers { permanent_uuid: "7397548a92234639b270117b2aa46274" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 43803 } }
I20260925 04:48:52.271974 22910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 920b12ae02fb43baa3f58ada6741ac04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920b12ae02fb43baa3f58ada6741ac04" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39091 } } peers { permanent_uuid: "428f0d0f5a9c4cc49873d4add804849d" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 40911 } } peers { permanent_uuid: "7397548a92234639b270117b2aa46274" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 43803 } } }
I20260925 04:48:52.272221 22910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 920b12ae02fb43baa3f58ada6741ac04 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:48:52.272843 22725 sys_catalog.cc:564] T 00000000000000000000000000000000 P 920b12ae02fb43baa3f58ada6741ac04 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:48:52.272928 22699 raft_consensus.cc:482] T 00000000000000000000000000000000 P 920b12ae02fb43baa3f58ada6741ac04 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 04:48:52.273028 22699 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 920b12ae02fb43baa3f58ada6741ac04 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:48:52.274451 22699 raft_consensus.cc:504] T 00000000000000000000000000000000 P 920b12ae02fb43baa3f58ada6741ac04 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920b12ae02fb43baa3f58ada6741ac04" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39091 } } peers { permanent_uuid: "428f0d0f5a9c4cc49873d4add804849d" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 40911 } } peers { permanent_uuid: "7397548a92234639b270117b2aa46274" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 43803 } }
I20260925 04:48:52.275497 22699 leader_election.cc:288] T 00000000000000000000000000000000 P 920b12ae02fb43baa3f58ada6741ac04 [CANDIDATE]: Term 1 election: Requested vote from peers 428f0d0f5a9c4cc49873d4add804849d (127.0.98.125:40911), 7397548a92234639b270117b2aa46274 (127.0.98.124:43803)
I20260925 04:48:52.276623 22774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "920b12ae02fb43baa3f58ada6741ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "428f0d0f5a9c4cc49873d4add804849d"
I20260925 04:48:52.276896 22774 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 428f0d0f5a9c4cc49873d4add804849d [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:48:52.278357 22774 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 428f0d0f5a9c4cc49873d4add804849d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 920b12ae02fb43baa3f58ada6741ac04 in term 1.
I20260925 04:48:52.278860 22849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "920b12ae02fb43baa3f58ada6741ac04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7397548a92234639b270117b2aa46274"
I20260925 04:48:52.279111 22849 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7397548a92234639b270117b2aa46274 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:48:52.280164 22849 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7397548a92234639b270117b2aa46274 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 920b12ae02fb43baa3f58ada6741ac04 in term 1.
I20260925 04:48:52.281087 22675 leader_election.cc:302] T 00000000000000000000000000000000 P 920b12ae02fb43baa3f58ada6741ac04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7397548a92234639b270117b2aa46274, 920b12ae02fb43baa3f58ada6741ac04; no voters:
I20260925 04:48:52.281761 22910 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 920b12ae02fb43baa3f58ada6741ac04 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:48:52.281991 22910 raft_consensus.cc:686] T 00000000000000000000000000000000 P 920b12ae02fb43baa3f58ada6741ac04 [term 1 LEADER]: Becoming Leader. State: Replica: 920b12ae02fb43baa3f58ada6741ac04, State: Running, Role: LEADER
I20260925 04:48:52.282501 22910 consensus_queue.cc:237] T 00000000000000000000000000000000 P 920b12ae02fb43baa3f58ada6741ac04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920b12ae02fb43baa3f58ada6741ac04" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39091 } } peers { permanent_uuid: "428f0d0f5a9c4cc49873d4add804849d" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 40911 } } peers { permanent_uuid: "7397548a92234639b270117b2aa46274" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 43803 } }
I20260925 04:48:52.284674 22910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 920b12ae02fb43baa3f58ada6741ac04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 920b12ae02fb43baa3f58ada6741ac04. Latest consensus state: current_term: 1 leader_uuid: "920b12ae02fb43baa3f58ada6741ac04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920b12ae02fb43baa3f58ada6741ac04" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39091 } } peers { permanent_uuid: "428f0d0f5a9c4cc49873d4add804849d" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 40911 } } peers { permanent_uuid: "7397548a92234639b270117b2aa46274" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 43803 } } }
I20260925 04:48:52.285312 22910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 920b12ae02fb43baa3f58ada6741ac04 [sys.catalog]: This master's current role is: LEADER
I20260925 04:48:52.285527 22657 master_runner.cc:263] No leader master found from master 920b12ae02fb43baa3f58ada6741ac04
I20260925 04:48:52.285584 22657 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 04:48:52.287691 22928 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:48:52.296533 22728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:52.285889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37936 (local address 127.0.98.126:39091)
0925 04:48:52.286010 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:52.286015 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:52.286442 (+ 427us) server_negotiation.cc:407] Connection header received
0925 04:48:52.286650 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:52.286653 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:52.286704 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:52.286759 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:52.287921 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:52.288772 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:52.290330 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:52.290655 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:52.290730 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:52.292676 (+ 1946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:52.292680 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:52.292683 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:48:52.292710 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:52.294530 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:52.295024 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:52.295028 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:52.295030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:52.295125 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:52.295445 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:52.295451 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:52.295456 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:52.295671 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:52.295800 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:52.296175 (+ 375us) server_negotiation.cc:299] Negotiation successful
0925 04:48:52.296332 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 04:48:52.300096 22928 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:48:52.302597 22880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:52.286594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38138 (local address 127.0.98.124:43803)
0925 04:48:52.286705 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:52.286709 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:52.287882 (+ 1173us) server_negotiation.cc:407] Connection header received
0925 04:48:52.288089 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:52.288092 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:52.288140 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:52.288190 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:52.289221 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:52.290049 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:52.294911 (+ 4862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:52.295266 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:52.295347 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:52.298432 (+ 3085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:52.298441 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:52.298444 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:48:52.298480 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:52.300282 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:52.300773 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:52.300790 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:52.300792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:52.300892 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:52.301453 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:52.301459 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:52.301464 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:52.301714 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:52.301847 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:52.302277 (+ 430us) server_negotiation.cc:299] Negotiation successful
0925 04:48:52.302409 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 04:48:52.304847 22805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:52.287292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57006 (local address 127.0.98.125:40911)
0925 04:48:52.287433 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:52.287437 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:52.291168 (+ 3731us) server_negotiation.cc:407] Connection header received
0925 04:48:52.291362 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:52.291366 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:52.291442 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:52.291498 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:52.292442 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:52.293240 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:52.297223 (+ 3983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:52.297552 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:52.297628 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:52.300971 (+ 3343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:52.300974 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:52.300977 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:48:52.301008 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:52.302756 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:52.303490 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:52.303494 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:52.303495 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:52.303581 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:52.303890 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:52.303893 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:52.303895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:52.304111 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:52.304227 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:52.304556 (+ 329us) server_negotiation.cc:299] Negotiation successful
0925 04:48:52.304676 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 04:48:52.317037 22849 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7397548a92234639b270117b2aa46274 [term 1 FOLLOWER]: Refusing update from remote peer 920b12ae02fb43baa3f58ada6741ac04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:48:52.317093 22774 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 428f0d0f5a9c4cc49873d4add804849d [term 1 FOLLOWER]: Refusing update from remote peer 920b12ae02fb43baa3f58ada6741ac04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:48:52.317788 22926 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 920b12ae02fb43baa3f58ada6741ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428f0d0f5a9c4cc49873d4add804849d" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 40911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:48:52.318500 22910 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 920b12ae02fb43baa3f58ada6741ac04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7397548a92234639b270117b2aa46274" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 43803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:48:52.323856 22883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7397548a92234639b270117b2aa46274 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 920b12ae02fb43baa3f58ada6741ac04. Latest consensus state: current_term: 1 leader_uuid: "920b12ae02fb43baa3f58ada6741ac04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920b12ae02fb43baa3f58ada6741ac04" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39091 } } peers { permanent_uuid: "428f0d0f5a9c4cc49873d4add804849d" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 40911 } } peers { permanent_uuid: "7397548a92234639b270117b2aa46274" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 43803 } } }
I20260925 04:48:52.323997 22883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7397548a92234639b270117b2aa46274 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:48:52.324605 22884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 428f0d0f5a9c4cc49873d4add804849d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 920b12ae02fb43baa3f58ada6741ac04. Latest consensus state: current_term: 1 leader_uuid: "920b12ae02fb43baa3f58ada6741ac04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920b12ae02fb43baa3f58ada6741ac04" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39091 } } peers { permanent_uuid: "428f0d0f5a9c4cc49873d4add804849d" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 40911 } } peers { permanent_uuid: "7397548a92234639b270117b2aa46274" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 43803 } } }
I20260925 04:48:52.324723 22884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 428f0d0f5a9c4cc49873d4add804849d [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:48:52.324721 22910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 920b12ae02fb43baa3f58ada6741ac04 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "920b12ae02fb43baa3f58ada6741ac04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920b12ae02fb43baa3f58ada6741ac04" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39091 } } peers { permanent_uuid: "428f0d0f5a9c4cc49873d4add804849d" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 40911 } } peers { permanent_uuid: "7397548a92234639b270117b2aa46274" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 43803 } } }
I20260925 04:48:52.324837 22910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 920b12ae02fb43baa3f58ada6741ac04 [sys.catalog]: This master's current role is: LEADER
I20260925 04:48:52.326341 22884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 428f0d0f5a9c4cc49873d4add804849d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "920b12ae02fb43baa3f58ada6741ac04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920b12ae02fb43baa3f58ada6741ac04" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39091 } } peers { permanent_uuid: "428f0d0f5a9c4cc49873d4add804849d" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 40911 } } peers { permanent_uuid: "7397548a92234639b270117b2aa46274" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 43803 } } }
I20260925 04:48:52.326455 22884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 428f0d0f5a9c4cc49873d4add804849d [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:48:52.326982 22933 mvcc.cc:205] Tried to move back new op lower bound from 7333116855562977280 to 7333116855434067968. Current Snapshot: MvccSnapshot[applied={T|T < 7333116855562977280}]
I20260925 04:48:52.328886 22934 mvcc.cc:205] Tried to move back new op lower bound from 7333116855562977280 to 7333116855434067968. Current Snapshot: MvccSnapshot[applied={T|T < 7333116855562977280}]
I20260925 04:48:52.333324 22928 catalog_manager.cc:1347] Generated new cluster ID: be18125c967946efb8b997acb8b84412
I20260925 04:48:52.333392 22928 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:48:52.340734 22926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 920b12ae02fb43baa3f58ada6741ac04 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "920b12ae02fb43baa3f58ada6741ac04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920b12ae02fb43baa3f58ada6741ac04" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39091 } } peers { permanent_uuid: "428f0d0f5a9c4cc49873d4add804849d" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 40911 } } peers { permanent_uuid: "7397548a92234639b270117b2aa46274" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 43803 } } }
I20260925 04:48:52.340886 22926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 920b12ae02fb43baa3f58ada6741ac04 [sys.catalog]: This master's current role is: LEADER
I20260925 04:48:52.342473 22883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7397548a92234639b270117b2aa46274 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "920b12ae02fb43baa3f58ada6741ac04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920b12ae02fb43baa3f58ada6741ac04" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39091 } } peers { permanent_uuid: "428f0d0f5a9c4cc49873d4add804849d" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 40911 } } peers { permanent_uuid: "7397548a92234639b270117b2aa46274" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 43803 } } }
I20260925 04:48:52.342603 22883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7397548a92234639b270117b2aa46274 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:48:52.352686 22928 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:48:52.354132 22928 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:48:52.358767 22936 mvcc.cc:205] Tried to move back new op lower bound from 7333116855562977280 to 7333116855434067968. Current Snapshot: MvccSnapshot[applied={T|T < 7333116855562977280}]
I20260925 04:48:52.392292 22928 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 920b12ae02fb43baa3f58ada6741ac04: Generated new TSK 0
I20260925 04:48:52.393201 22928 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 04:48:52.433277 22805 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:48:52.429978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57020 (local address 127.0.98.125:40911)
0925 04:48:52.430119 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:52.430124 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:52.432116 (+ 1992us) server_negotiation.cc:407] Connection header received
0925 04:48:52.432332 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:52.432335 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:52.432390 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:52.432446 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:52.433141 (+ 695us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57020: BlockingRecv error: recv got EOF from 127.0.0.1:57020 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260925 04:48:52.434800 22880 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:48:52.430862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38152 (local address 127.0.98.124:43803)
0925 04:48:52.431001 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:52.431006 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:52.433808 (+ 2802us) server_negotiation.cc:407] Connection header received
0925 04:48:52.433926 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:52.433929 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:52.433985 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:52.434083 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:52.434652 (+ 569us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38152: BlockingRecv error: recv got EOF from 127.0.0.1:38152 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260925 04:48:52.436179 22728 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:48:52.431536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37952 (local address 127.0.98.126:39091)
0925 04:48:52.431666 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:52.431673 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:52.434173 (+ 2500us) server_negotiation.cc:407] Connection header received
0925 04:48:52.435378 (+ 1205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:52.435382 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:52.435438 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:52.435495 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:52.436059 (+ 564us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37952: BlockingRecv error: recv got EOF from 127.0.0.1:37952 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260925 04:48:52.441064 22805 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:48:52.439480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57022 (local address 127.0.98.125:40911)
0925 04:48:52.439596 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:52.439601 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:52.440062 (+ 461us) server_negotiation.cc:407] Connection header received
0925 04:48:52.440274 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:52.440277 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:52.440355 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:52.440414 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:52.440942 (+ 528us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57022: BlockingRecv error: recv got EOF from 127.0.0.1:57022 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260925 04:48:52.446403 22805 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:48:52.444594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57036 (local address 127.0.98.125:40911)
0925 04:48:52.444693 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:52.444697 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:52.445245 (+ 548us) server_negotiation.cc:407] Connection header received
0925 04:48:52.445426 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:52.445429 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:52.445477 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:52.445543 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:52.446297 (+ 754us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57036: BlockingRecv error: recv got EOF from 127.0.0.1:57036 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260925 04:48:52.452106 22805 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:48:52.450428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57052 (local address 127.0.98.125:40911)
0925 04:48:52.450532 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:52.450537 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:52.451122 (+ 585us) server_negotiation.cc:407] Connection header received
0925 04:48:52.451304 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:52.451307 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:52.451362 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:52.451420 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:52.451993 (+ 573us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57052: BlockingRecv error: recv got EOF from 127.0.0.1:57052 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260925 04:48:52.460145 22805 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:48:52.458439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57056 (local address 127.0.98.125:40911)
0925 04:48:52.458578 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:52.458583 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:52.459154 (+ 571us) server_negotiation.cc:407] Connection header received
0925 04:48:52.459354 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:52.459357 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:52.459413 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:52.459469 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:52.460008 (+ 539us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57056: BlockingRecv error: recv got EOF from 127.0.0.1:57056 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260925 04:48:52.472512 22805 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:48:52.470664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57066 (local address 127.0.98.125:40911)
0925 04:48:52.470778 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:52.470783 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:52.471381 (+ 598us) server_negotiation.cc:407] Connection header received
0925 04:48:52.471650 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:52.471653 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:52.471710 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:52.471768 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:52.472382 (+ 614us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57066: BlockingRecv error: recv got EOF from 127.0.0.1:57066 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260925 04:48:52.492218 22805 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:48:52.490912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57080 (local address 127.0.98.125:40911)
0925 04:48:52.491032 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:52.491037 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:52.491322 (+ 285us) server_negotiation.cc:407] Connection header received
0925 04:48:52.491451 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:52.491454 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:52.491521 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:52.491618 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:52.492094 (+ 476us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57080: BlockingRecv error: recv got EOF from 127.0.0.1:57080 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260925 04:48:52.527426 22805 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:48:52.526302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57090 (local address 127.0.98.125:40911)
0925 04:48:52.526470 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:52.526474 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:52.526494 (+ 20us) server_negotiation.cc:407] Connection header received
0925 04:48:52.526637 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:52.526640 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:52.526726 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:52.526821 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:52.527313 (+ 492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57090: BlockingRecv error: recv got EOF from 127.0.0.1:57090 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260925 04:48:52.594797 22805 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:48:52.593628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57096 (local address 127.0.98.125:40911)
0925 04:48:52.593803 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:52.593807 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:52.593866 (+ 59us) server_negotiation.cc:407] Connection header received
0925 04:48:52.594001 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:52.594004 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:52.594050 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:52.594126 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:52.594638 (+ 512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57096: BlockingRecv error: recv got EOF from 127.0.0.1:57096 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260925 04:48:52.725987 22805 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:48:52.724851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57104 (local address 127.0.98.125:40911)
0925 04:48:52.724992 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:52.724997 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:52.725130 (+ 133us) server_negotiation.cc:407] Connection header received
0925 04:48:52.725278 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:52.725281 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:52.725336 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:52.725435 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:52.725851 (+ 416us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57104: BlockingRecv error: recv got EOF from 127.0.0.1:57104 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260925 04:48:52.985592 22805 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:48:52.984341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57108 (local address 127.0.98.125:40911)
0925 04:48:52.984473 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:52.984480 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:52.984668 (+ 188us) server_negotiation.cc:407] Connection header received
0925 04:48:52.984765 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:52.984768 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:52.984820 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:52.984910 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:52.985456 (+ 546us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57108: BlockingRecv error: recv got EOF from 127.0.0.1:57108 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 04:48:53.258072 22895 catalog_manager.cc:1259] Loaded cluster ID: be18125c967946efb8b997acb8b84412
I20260925 04:48:53.258157 22895 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7397548a92234639b270117b2aa46274: loading cluster ID for follower catalog manager: success
I20260925 04:48:53.260036 22895 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7397548a92234639b270117b2aa46274: acquiring CA information for follower catalog manager: success
I20260925 04:48:53.260485 22906 catalog_manager.cc:1259] Loaded cluster ID: be18125c967946efb8b997acb8b84412
I20260925 04:48:53.260567 22906 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 428f0d0f5a9c4cc49873d4add804849d: loading cluster ID for follower catalog manager: success
I20260925 04:48:53.261346 22895 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7397548a92234639b270117b2aa46274: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 04:48:53.262130 22906 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 428f0d0f5a9c4cc49873d4add804849d: acquiring CA information for follower catalog manager: success
I20260925 04:48:53.263159 22906 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 428f0d0f5a9c4cc49873d4add804849d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 04:48:53.503549 23007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:53.499732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57120 (local address 127.0.98.125:40911)
0925 04:48:53.499999 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:53.500004 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:53.500106 (+ 102us) server_negotiation.cc:407] Connection header received
0925 04:48:53.500234 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:53.500237 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:53.500361 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:53.500475 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:48:53.500716 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:53.501249 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:53.502065 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:53.502347 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:53.502440 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:53.502704 (+ 264us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:48:53.503134 (+ 430us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:48:53.503224 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:53.503281 (+ 57us) server_negotiation.cc:299] Negotiation successful
0925 04:48:53.503330 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":74,"threads_started":1}
I20260925 04:48:53.509932 23013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:53.506707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38156 (local address 127.0.98.124:43803)
0925 04:48:53.506933 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:53.506936 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:53.506950 (+ 14us) server_negotiation.cc:407] Connection header received
0925 04:48:53.507045 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:53.507048 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:53.507155 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:53.507247 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:48:53.507417 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:53.507902 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:53.508677 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:53.508887 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:53.508964 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:53.509202 (+ 238us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:48:53.509610 (+ 408us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:48:53.509702 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:53.509758 (+ 56us) server_negotiation.cc:299] Negotiation successful
0925 04:48:53.509816 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":66,"threads_started":1}
I20260925 04:48:53.517017 23007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:53.513020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57134 (local address 127.0.98.125:40911)
0925 04:48:53.513145 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:53.513150 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:53.513528 (+ 378us) server_negotiation.cc:407] Connection header received
0925 04:48:53.513639 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:53.513643 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:53.513718 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:53.513793 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:48:53.514050 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:53.514876 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:53.515803 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:53.516141 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:53.516240 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:53.516511 (+ 271us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:48:53.516603 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:48:53.516668 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:53.516758 (+ 90us) server_negotiation.cc:299] Negotiation successful
0925 04:48:53.516808 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 04:48:53.517140 23013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:53.513051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38166 (local address 127.0.98.124:43803)
0925 04:48:53.513242 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:53.513248 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:53.513534 (+ 286us) server_negotiation.cc:407] Connection header received
0925 04:48:53.513740 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:53.513746 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:53.513802 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:53.513867 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:48:53.514243 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:53.514764 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:53.515944 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:53.516274 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:53.516387 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:53.516643 (+ 256us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:48:53.516723 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:48:53.516799 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:53.516969 (+ 170us) server_negotiation.cc:299] Negotiation successful
0925 04:48:53.517023 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260925 04:48:53.517844 23021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:53.513171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37954 (local address 127.0.98.126:39091)
0925 04:48:53.513453 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:53.513459 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:53.513476 (+ 17us) server_negotiation.cc:407] Connection header received
0925 04:48:53.513570 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:53.513573 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:53.513743 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:53.513875 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:48:53.514294 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:53.515092 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:53.516147 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:53.516543 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:53.516655 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:53.517092 (+ 437us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:48:53.517463 (+ 371us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:48:53.517554 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:53.517624 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 04:48:53.517680 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":79,"threads_started":1}
I20260925 04:48:53.521536 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 22657
I20260925 04:48:53.528136 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 22732
I20260925 04:48:53.533888 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 22806
2026-09-25T04:48:53Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3170 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311449873301-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:48:53 dist-test-slave-kghz krb5kdc[23024](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:48:53 dist-test-slave-kghz krb5kdc[23024](info): set up 2 sockets
Sep 25 04:48:53 dist-test-slave-kghz krb5kdc[23024](info): commencing operation
krb5kdc: starting...
W20260925 04:48:54.586799 393 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311449873301-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311449873301-393-0/krb5kdc/test-user.keytab.
Sep 25 04:48:54 dist-test-slave-kghz krb5kdc[23024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:48:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:48:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260925 04:48:54.714206 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311449873301-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311449873301-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311449873301-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:42903
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:39149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:42903,127.0.98.125:36771,127.0.98.124:40191
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:48:54.824673 23037 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 04:48:54.825057 23037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:48:54.825133 23037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:48:54.825217 23037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:48:54.829336 23037 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:48:54.829447 23037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:48:54.829483 23037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:48:54.829525 23037 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:48:54.829560 23037 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:48:54.834904 23037 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:39149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311449873301-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311449873301-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:42903,127.0.98.125:36771,127.0.98.124:40191
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311449873301-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:42903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23037
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311449873301-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:48:54.836026 23037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:48:54.837074 23037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:48:54.845587 23043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:48:54.845621 23037 server_base.cc:1034] running on GCE node
W20260925 04:48:54.845656 23042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:48:54.845675 23045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:48:54.846472 23037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:48:54.847528 23037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:48:54.848701 23037 hybrid_clock.cc:648] HybridClock initialized: now 1790311734848707 us; error 47 us; skew 500 ppm
Sep 25 04:48:54 dist-test-slave-kghz krb5kdc[23024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311734, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:48:54.851727 23037 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260925 04:48:54.852689 23037 webserver.cc:460] Webserver started at http://127.0.98.126:42625/ using document root <none> and password file <none>
I20260925 04:48:54.853255 23037 fs_manager.cc:359] Metadata directory not provided
I20260925 04:48:54.853346 23037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:48:54.853597 23037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:48:54.855736 23037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311449873301-393-0/minicluster-data/master-0/data/instance:
uuid: "1268f111a3bd40469d44908cf343a0db"
format_stamp: "Formatted at 2026-09-25 04:48:54 on dist-test-slave-kghz"
server_key: "c6b4d9a4847c97f7c409042bd507d00a"
server_key_iv: "112cc3e1967254a7952b76d5900f2901"
server_key_version: "encryptionkey@0"
I20260925 04:48:54.856309 23037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311449873301-393-0/minicluster-data/master-0/wal/instance:
uuid: "1268f111a3bd40469d44908cf343a0db"
format_stamp: "Formatted at 2026-09-25 04:48:54 on dist-test-slave-kghz"
server_key: "c6b4d9a4847c97f7c409042bd507d00a"
server_key_iv: "112cc3e1967254a7952b76d5900f2901"
server_key_version: "encryptionkey@0"
I20260925 04:48:54.860142 23037 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 04:48:54.862648 23052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:48:54.863883 23037 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 04:48:54.864029 23037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311449873301-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311449873301-393-0/minicluster-data/master-0/wal
uuid: "1268f111a3bd40469d44908cf343a0db"
format_stamp: "Formatted at 2026-09-25 04:48:54 on dist-test-slave-kghz"
server_key: "c6b4d9a4847c97f7c409042bd507d00a"
server_key_iv: "112cc3e1967254a7952b76d5900f2901"
server_key_version: "encryptionkey@0"
I20260925 04:48:54.864146 23037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311449873301-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311449873301-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311449873301-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:48:54.875278 23037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:48:54.878224 23037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:48:54.878481 23037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:48:54.885740 23037 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:42903
I20260925 04:48:54.886061 23104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:42903 every 8 connection(s)
I20260925 04:48:54.886783 23037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311449873301-393-0/minicluster-data/master-0/data/info.pb
I20260925 04:48:54.889711 23105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:48:54.890905 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 23037
I20260925 04:48:54.891068 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311449873301-393-0/minicluster-data/master-0/wal/instance
I20260925 04:48:54.891371 393 external_mini_cluster.cc:1428] Setting key ec9ef38eae56bdddee232e01ff2dfa20
I20260925 04:48:54.893904 23105 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42903 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.125@KRBTEST.COM" created.
Sep 25 04:48:54 dist-test-slave-kghz krb5kdc[23024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311734, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:48:54.911227 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:54.897457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42676 (local address 127.0.98.126:42903)
0925 04:48:54.897836 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:54.897848 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:54.897896 (+ 48us) server_negotiation.cc:407] Connection header received
0925 04:48:54.898453 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:54.898459 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:54.898760 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:54.898980 (+ 220us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:54.899933 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:54.900943 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:54.902133 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:54.902807 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:54.902947 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:54.905682 (+ 2735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:54.905692 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:54.905710 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:48:54.905742 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:54.908006 (+ 2264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:54.908524 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:54.908532 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:54.908536 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:54.908695 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:54.909033 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:54.909040 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:54.909043 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:54.909594 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:54.909740 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:54.910145 (+ 405us) server_negotiation.cc:299] Negotiation successful
0925 04:48:54.910492 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":138,"server-negotiator.queue_time_us":233,"thread_start_us":98,"threads_started":1}
I20260925 04:48:54.911227 23108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:54.896536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:42903 (local address 127.0.0.1:42676)
0925 04:48:54.897037 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:54.897091 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 04:48:54.898075 (+ 984us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:48:54.898994 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:48:54.898998 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:48:54.899439 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:48:54.899711 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:54.899725 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:54.901095 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:54.901098 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:54.901969 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:54.901979 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:54.902983 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:54.902990 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:54.903130 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:54.903808 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:48:54.903886 (+ 78us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:48:54.905524 (+ 1638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:48:54.908149 (+ 2625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:48:54.908158 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:48:54.908166 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:48:54.908402 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:48:54.908808 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:48:54.908810 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:48:54.908813 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:48:54.908937 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:48:54.909730 (+ 793us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:48:54.909735 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:48:54.909875 (+ 140us) client_negotiation.cc:769] Sending connection context
0925 04:48:54.910020 (+ 145us) client_negotiation.cc:240] Negotiation successful
0925 04:48:54.910362 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":135,"threads_started":1}
I20260925 04:48:54.913895 23105 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.125" port: 36771 } has no permanent_uuid. Determining permanent_uuid...
I20260925 04:48:54.914846 23108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:54.914545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:36771 (local address 127.0.0.1:51722)
0925 04:48:54.914665 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:54.914774 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.125:36771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 04:48:54.915052 23055 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.125:36771: connect: Connection refused (error 111)
I20260925 04:48:54.915946 23108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:54.915723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:36771 (local address 127.0.0.1:51738)
0925 04:48:54.915834 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:54.915886 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.125:36771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 04:48:54.916133 23105 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.125:36771: Network error: Client connection negotiation failed: client connection to 127.0.98.125:36771: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311449873301-393-0/krb5kdc/kudu_127.0.98.125.keytab.
Entry for principal kudu/127.0.98.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311449873301-393-0/krb5kdc/kudu_127.0.98.125.keytab.
I20260925 04:48:54.922195 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311449873301-393-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311449873301-393-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311449873301-393-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311449873301-393-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.125:36771
--webserver_interface=127.0.98.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311449873301-393-0/krb5kdc/kudu_127.0.98.125.keytab
--principal=kudu/127.0.98.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:39149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:42903,127.0.98.125:36771,127.0.98.124:40191
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 04:48:54.965411 23105 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.125" port: 36771 } attempt: 1
I20260925 04:48:54.966465 23108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:54.966090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:36771 (local address 127.0.0.1:51740)
0925 04:48:54.966317 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:54.966383 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.125:36771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260925 04:48:54.967204 23108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:54.966974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:36771 (local address 127.0.0.1:51746)
0925 04:48:54.967092 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:54.967146 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.125:36771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 04:48:54.967345 23105 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.125:36771: Network error: Client connection negotiation failed: client connection to 127.0.98.125:36771: connect: Connection refused (error 111)
W20260925 04:48:55.035277 23112 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
I20260925 04:48:55.035521 23105 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.125" port: 36771 } attempt: 2
W20260925 04:48:55.035645 23112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:48:55.035692 23112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:48:55.035779 23112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260925 04:48:55.036464 23108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:55.036111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:36771 (local address 127.0.0.1:51760)
0925 04:48:55.036304 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:55.036375 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.125:36771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260925 04:48:55.037338 23108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:55.037107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:36771 (local address 127.0.0.1:51774)
0925 04:48:55.037214 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:55.037276 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.125:36771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260925 04:48:55.037468 23105 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.125:36771: Network error: Client connection negotiation failed: client connection to 127.0.98.125:36771: connect: Connection refused (error 111)
W20260925 04:48:55.040364 23112 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:48:55.040419 23112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:48:55.040439 23112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:48:55.040455 23112 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:48:55.040472 23112 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:48:55.045809 23112 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:39149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311449873301-393-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311449873301-393-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:42903,127.0.98.125:36771,127.0.98.124:40191
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311449873301-393-0/krb5kdc/kudu_127.0.98.125.keytab
--principal=kudu/127.0.98.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.125:36771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311449873301-393-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23112
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311449873301-393-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:48:55.046922 23112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:48:55.047930 23112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:48:55.056903 23118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:48:55.056934 23120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:48:55.056903 23117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:48:55.057166 23112 server_base.cc:1034] running on GCE node
I20260925 04:48:55.057858 23112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:48:55.059137 23112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:48:55.060307 23112 hybrid_clock.cc:648] HybridClock initialized: now 1790311735060290 us; error 31 us; skew 500 ppm
Sep 25 04:48:55 dist-test-slave-kghz krb5kdc[23024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311735, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:48:55.063289 23112 init.cc:377] Logged in from keytab as kudu/127.0.98.125@KRBTEST.COM (short username kudu)
I20260925 04:48:55.064299 23112 webserver.cc:460] Webserver started at http://127.0.98.125:40799/ using document root <none> and password file <none>
I20260925 04:48:55.064867 23112 fs_manager.cc:359] Metadata directory not provided
I20260925 04:48:55.064968 23112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:48:55.065202 23112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:48:55.067389 23112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311449873301-393-0/minicluster-data/master-1/data/instance:
uuid: "b2d4802490b74daf939925e65397742c"
format_stamp: "Formatted at 2026-09-25 04:48:55 on dist-test-slave-kghz"
server_key: "92c2d56da6abf1ef6f833e83d002a547"
server_key_iv: "2c14db4809885105ff5c70c8594b26fc"
server_key_version: "encryptionkey@0"
I20260925 04:48:55.067965 23112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311449873301-393-0/minicluster-data/master-1/wal/instance:
uuid: "b2d4802490b74daf939925e65397742c"
format_stamp: "Formatted at 2026-09-25 04:48:55 on dist-test-slave-kghz"
server_key: "92c2d56da6abf1ef6f833e83d002a547"
server_key_iv: "2c14db4809885105ff5c70c8594b26fc"
server_key_version: "encryptionkey@0"
I20260925 04:48:55.071816 23112 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 04:48:55.074541 23127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:48:55.075779 23112 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:48:55.075932 23112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311449873301-393-0/minicluster-data/master-1/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311449873301-393-0/minicluster-data/master-1/wal
uuid: "b2d4802490b74daf939925e65397742c"
format_stamp: "Formatted at 2026-09-25 04:48:55 on dist-test-slave-kghz"
server_key: "92c2d56da6abf1ef6f833e83d002a547"
server_key_iv: "2c14db4809885105ff5c70c8594b26fc"
server_key_version: "encryptionkey@0"
I20260925 04:48:55.076049 23112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311449873301-393-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311449873301-393-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311449873301-393-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:48:55.092291 23112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:48:55.095338 23112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:48:55.095572 23112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:48:55.102890 23112 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.125:36771
I20260925 04:48:55.103180 23179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.125:36771 every 8 connection(s)
I20260925 04:48:55.103878 23112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311449873301-393-0/minicluster-data/master-1/data/info.pb
I20260925 04:48:55.106712 23180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:48:55.109261 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 23112
I20260925 04:48:55.109409 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311449873301-393-0/minicluster-data/master-1/wal/instance
I20260925 04:48:55.109719 393 external_mini_cluster.cc:1428] Setting key b8e8ff478c81dbc545a914a9fa288f6d
I20260925 04:48:55.111052 23180 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42903 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.124@KRBTEST.COM" created.
Sep 25 04:48:55 dist-test-slave-kghz krb5kdc[23024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311735, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.125@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:48:55.126279 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:55.113201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42690 (local address 127.0.98.126:42903)
0925 04:48:55.113370 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:55.113375 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:55.114429 (+ 1054us) server_negotiation.cc:407] Connection header received
0925 04:48:55.115425 (+ 996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:55.115431 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:55.115513 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:55.115632 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:55.116837 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:55.117754 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:55.119100 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:55.119336 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:55.119424 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:55.122038 (+ 2614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:55.122041 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:55.122043 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:48:55.122067 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:55.124083 (+ 2016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:55.124645 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:55.124649 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:55.124651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:55.124734 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:55.125078 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:55.125082 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:55.125084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:55.125282 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:55.125392 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:55.125937 (+ 545us) server_negotiation.cc:299] Negotiation successful
0925 04:48:55.126070 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 04:48:55.127079 23183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:55.113661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:42903 (local address 127.0.0.1:42690)
0925 04:48:55.114184 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:55.114240 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 04:48:55.115197 (+ 957us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:48:55.115793 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:48:55.115799 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:48:55.116198 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:48:55.116674 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:55.116687 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:55.117959 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:55.117963 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:55.118979 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:55.118994 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:55.119474 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:55.119476 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:55.119617 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:55.120283 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:48:55.120301 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:48:55.121850 (+ 1549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:48:55.124229 (+ 2379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:48:55.124235 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:48:55.124251 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:48:55.124534 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:48:55.124828 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:48:55.124832 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:48:55.124834 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:48:55.124980 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:48:55.125389 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:48:55.125406 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:48:55.125672 (+ 266us) client_negotiation.cc:769] Sending connection context
0925 04:48:55.125873 (+ 201us) client_negotiation.cc:240] Negotiation successful
0925 04:48:55.126210 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":140,"threads_started":1}
I20260925 04:48:55.128446 23180 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.125" port: 36771 } has no permanent_uuid. Determining permanent_uuid...
I20260925 04:48:55.128621 23105 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.125" port: 36771 } attempt: 3
Sep 25 04:48:55 dist-test-slave-kghz krb5kdc[23024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311735, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.125@KRBTEST.COM for kudu/127.0.98.125@KRBTEST.COM
Sep 25 04:48:55 dist-test-slave-kghz krb5kdc[23024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311734, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for kudu/127.0.98.125@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311449873301-393-0/krb5kdc/kudu_127.0.98.124.keytab.
Entry for principal kudu/127.0.98.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311449873301-393-0/krb5kdc/kudu_127.0.98.124.keytab.
I20260925 04:48:55.138701 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHechoP/build/debug/bin/kudu
/tmp/dist-test-taskHechoP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311449873301-393-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311449873301-393-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311449873301-393-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311449873301-393-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.124:40191
--webserver_interface=127.0.98.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311449873301-393-0/krb5kdc/kudu_127.0.98.124.keytab
--principal=kudu/127.0.98.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:39149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:42903,127.0.98.125:36771,127.0.98.124:40191
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311449873301-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311449873301-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311449873301-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 04:48:55.140754 23108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:55.129062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:36771 (local address 127.0.0.1:51792)
0925 04:48:55.129184 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:55.129199 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:48:55.129304 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:48:55.130059 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:48:55.130062 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:48:55.130301 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:48:55.130439 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:55.130450 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:55.131518 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:55.131523 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:55.132121 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:55.132130 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:55.132880 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:55.132883 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:55.133003 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:55.133705 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:48:55.133724 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:48:55.135471 (+ 1747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:48:55.138748 (+ 3277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:48:55.138753 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:48:55.138756 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:48:55.139046 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:48:55.139385 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:48:55.139388 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:48:55.139391 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:48:55.139507 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:48:55.140140 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:48:55.140147 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:48:55.140286 (+ 139us) client_negotiation.cc:769] Sending connection context
0925 04:48:55.140377 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 04:48:55.140567 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 04:48:55.140764 23183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:55.128982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:36771 (local address 127.0.0.1:51786)
0925 04:48:55.129465 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:55.129479 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:48:55.129566 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:48:55.130019 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:48:55.130021 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:48:55.130305 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:48:55.130414 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:55.130420 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:55.131484 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:55.131490 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:55.132077 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:55.132085 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:55.132879 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:55.132883 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:55.133094 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:55.133504 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:48:55.133516 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:48:55.135044 (+ 1528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:48:55.139139 (+ 4095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:48:55.139145 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:48:55.139150 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:48:55.139426 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:48:55.139746 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:48:55.139752 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:48:55.139754 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:48:55.139836 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:48:55.140300 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:48:55.140303 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:48:55.140390 (+ 87us) client_negotiation.cc:769] Sending connection context
0925 04:48:55.140489 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 04:48:55.140620 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"mutex_wait_us":16}
I20260925 04:48:55.141179 23185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:55.129078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51786 (local address 127.0.98.125:36771)
0925 04:48:55.129446 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:55.129471 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:55.129670 (+ 199us) server_negotiation.cc:407] Connection header received
0925 04:48:55.129699 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:55.129702 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:55.129875 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:55.129980 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:55.130528 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:55.131370 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:55.132193 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:55.132764 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:55.132881 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:55.135193 (+ 2312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:55.135639 (+ 446us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:55.135663 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:48:55.135692 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:55.138980 (+ 3288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:55.139554 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:55.139559 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:55.139562 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:55.139646 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:55.139936 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:55.139940 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:55.139942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:55.140205 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:55.140356 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:55.140614 (+ 258us) server_negotiation.cc:299] Negotiation successful
0925 04:48:55.140792 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":257,"spinlock_wait_cycles":23680,"thread_start_us":142,"threads_started":1}
I20260925 04:48:55.141582 23186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:55.129442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51792 (local address 127.0.98.125:36771)
0925 04:48:55.129682 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:55.129690 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:55.129703 (+ 13us) server_negotiation.cc:407] Connection header received
0925 04:48:55.129740 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:55.129742 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:55.129877 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:55.130012 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:55.130585 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:55.131412 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:55.132238 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:55.132760 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:55.132878 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:55.135676 (+ 2798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:55.135679 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:55.135681 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:48:55.135697 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:55.138555 (+ 2858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:55.139195 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:55.139200 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:55.139201 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:55.139279 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:55.139596 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:55.139602 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:55.139606 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:55.139987 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:55.141247 (+ 1260us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:55.141336 (+ 89us) server_negotiation.cc:299] Negotiation successful
0925 04:48:55.141442 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":329,"server-negotiator.queue_time_us":181,"spinlock_wait_cycles":9344,"thread_start_us":88,"threads_started":1}
I20260925 04:48:55.143153 23180 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.124" port: 40191 } has no permanent_uuid. Determining permanent_uuid...
I20260925 04:48:55.143312 23105 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.124" port: 40191 } has no permanent_uuid. Determining permanent_uuid...
I20260925 04:48:55.144066 23108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:55.143797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:40191 (local address 127.0.0.1:48860)
0925 04:48:55.143926 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:55.143983 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:40191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260925 04:48:55.144232 23183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:55.143814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:40191 (local address 127.0.0.1:48858)
0925 04:48:55.144060 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:55.144147 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:40191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149}
W20260925 04:48:55.144408 23130 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.124:40191: connect: Connection refused (error 111)
I20260925 04:48:55.144783 23108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:55.144539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:40191 (local address 127.0.0.1:48878)
0925 04:48:55.144666 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:55.144717 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:40191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260925 04:48:55.144945 23105 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.124:40191: Network error: Client connection negotiation failed: client connection to 127.0.98.124:40191: connect: Connection refused (error 111)
I20260925 04:48:55.145423 23189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:55.145102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:40191 (local address 127.0.0.1:48886)
0925 04:48:55.145311 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:55.145353 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:40191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":101,"threads_started":1}
W20260925 04:48:55.145637 23180 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.124:40191: Network error: Client connection negotiation failed: client connection to 127.0.98.124:40191: connect: Connection refused (error 111)
I20260925 04:48:55.176098 23105 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.124" port: 40191 } attempt: 1
I20260925 04:48:55.176980 23108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:55.176681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:40191 (local address 127.0.0.1:48902)
0925 04:48:55.176838 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:55.176899 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:40191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260925 04:48:55.177721 23108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:55.177485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:40191 (local address 127.0.0.1:48910)
0925 04:48:55.177612 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:55.177667 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:40191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260925 04:48:55.177851 23105 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.124:40191: Network error: Client connection negotiation failed: client connection to 127.0.98.124:40191: connect: Connection refused (error 111)
I20260925 04:48:55.194888 23180 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.124" port: 40191 } attempt: 1
I20260925 04:48:55.195793 23189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:55.195493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:40191 (local address 127.0.0.1:48916)
0925 04:48:55.195650 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:55.195715 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:40191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260925 04:48:55.196623 23189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:55.196423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:40191 (local address 127.0.0.1:48918)
0925 04:48:55.196527 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:55.196571 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:40191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260925 04:48:55.196758 23180 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.124:40191: Network error: Client connection negotiation failed: client connection to 127.0.98.124:40191: connect: Connection refused (error 111)
I20260925 04:48:55.253052 23105 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.124" port: 40191 } attempt: 2
I20260925 04:48:55.253918 23108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:55.253604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:40191 (local address 127.0.0.1:48928)
0925 04:48:55.253769 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:55.253838 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:40191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260925 04:48:55.254733 23108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:55.254508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:40191 (local address 127.0.0.1:48938)
0925 04:48:55.254620 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:55.254680 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:40191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 04:48:55.254922 23105 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.124:40191: Network error: Client connection negotiation failed: client connection to 127.0.98.124:40191: connect: Connection refused (error 111)
W20260925 04:48:55.254895 23187 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 04:48:55.255311 23187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:48:55.255384 23187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:48:55.255458 23187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:48:55.259881 23187 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:48:55.259987 23187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:48:55.260025 23187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:48:55.260061 23187 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:48:55.260095 23187 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:48:55.264945 23180 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.124" port: 40191 } attempt: 2
I20260925 04:48:55.265894 23189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:55.265567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:40191 (local address 127.0.0.1:48952)
0925 04:48:55.265747 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:55.265816 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:40191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260925 04:48:55.266116 23187 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:39149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311449873301-393-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311449873301-393-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:42903,127.0.98.125:36771,127.0.98.124:40191
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311449873301-393-0/krb5kdc/kudu_127.0.98.124.keytab
--principal=kudu/127.0.98.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.124:40191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311449873301-393-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311449873301-393-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:48:55.266824 23189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:55.266574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:40191 (local address 127.0.0.1:48954)
0925 04:48:55.266699 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:55.266758 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:40191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260925 04:48:55.267040 23180 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.124:40191: Network error: Client connection negotiation failed: client connection to 127.0.98.124:40191: connect: Connection refused (error 111)
I20260925 04:48:55.267549 23187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:48:55.268707 23187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:48:55.277825 23197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:48:55.277843 23195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:48:55.277825 23194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:48:55.278087 23187 server_base.cc:1034] running on GCE node
I20260925 04:48:55.279202 23187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:48:55.280426 23187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:48:55.281612 23187 hybrid_clock.cc:648] HybridClock initialized: now 1790311735281571 us; error 62 us; skew 500 ppm
Sep 25 04:48:55 dist-test-slave-kghz krb5kdc[23024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311735, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:48:55.284524 23187 init.cc:377] Logged in from keytab as kudu/127.0.98.124@KRBTEST.COM (short username kudu)
I20260925 04:48:55.285406 23187 webserver.cc:460] Webserver started at http://127.0.98.124:38325/ using document root <none> and password file <none>
I20260925 04:48:55.286010 23187 fs_manager.cc:359] Metadata directory not provided
I20260925 04:48:55.286074 23187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:48:55.286286 23187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:48:55.288378 23187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311449873301-393-0/minicluster-data/master-2/data/instance:
uuid: "cd77afae8b7d417a8896d0f57f48ea2d"
format_stamp: "Formatted at 2026-09-25 04:48:55 on dist-test-slave-kghz"
server_key: "bf5eff05be0cb1e879dd635fa94dbd41"
server_key_iv: "57563653ebdd8c6f10c024b5a2f5fd9d"
server_key_version: "encryptionkey@0"
I20260925 04:48:55.288930 23187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311449873301-393-0/minicluster-data/master-2/wal/instance:
uuid: "cd77afae8b7d417a8896d0f57f48ea2d"
format_stamp: "Formatted at 2026-09-25 04:48:55 on dist-test-slave-kghz"
server_key: "bf5eff05be0cb1e879dd635fa94dbd41"
server_key_iv: "57563653ebdd8c6f10c024b5a2f5fd9d"
server_key_version: "encryptionkey@0"
I20260925 04:48:55.292872 23187 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260925 04:48:55.295694 23204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:48:55.297219 23187 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 04:48:55.297363 23187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311449873301-393-0/minicluster-data/master-2/data,/tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311449873301-393-0/minicluster-data/master-2/wal
uuid: "cd77afae8b7d417a8896d0f57f48ea2d"
format_stamp: "Formatted at 2026-09-25 04:48:55 on dist-test-slave-kghz"
server_key: "bf5eff05be0cb1e879dd635fa94dbd41"
server_key_iv: "57563653ebdd8c6f10c024b5a2f5fd9d"
server_key_version: "encryptionkey@0"
I20260925 04:48:55.297493 23187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311449873301-393-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311449873301-393-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311449873301-393-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:48:55.312295 23187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:48:55.316375 23187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:48:55.316617 23187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:48:55.324402 23187 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.124:40191
I20260925 04:48:55.324712 23256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.124:40191 every 8 connection(s)
I20260925 04:48:55.325528 23187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311449873301-393-0/minicluster-data/master-2/data/info.pb
I20260925 04:48:55.326502 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHechoP/build/debug/bin/kudu as pid 23187
I20260925 04:48:55.326638 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHechoP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790311449873301-393-0/minicluster-data/master-2/wal/instance
I20260925 04:48:55.326920 393 external_mini_cluster.cc:1428] Setting key 9574d52f94269bc253f749758367976b
I20260925 04:48:55.328981 23257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:48:55.333338 23257 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42903 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 04:48:55 dist-test-slave-kghz krb5kdc[23024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.124@KRBTEST.COM
Sep 25 04:48:55 dist-test-slave-kghz krb5kdc[23024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311735, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.124@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260925 04:48:55.342700 23260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:55.328476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48956 (local address 127.0.98.124:40191)
0925 04:48:55.328944 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:55.328960 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:55.328998 (+ 38us) server_negotiation.cc:407] Connection header received
0925 04:48:55.329705 (+ 707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:55.329722 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:55.330065 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:55.330456 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:55.331285 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:55.332303 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:55.333498 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:55.334067 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:55.334202 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:55.337145 (+ 2943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:55.337150 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:55.337159 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:48:55.337185 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:55.339311 (+ 2126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:55.339872 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:55.339877 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:55.339882 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:55.340036 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:55.340340 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:55.340342 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:55.340344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:55.340894 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:55.341142 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:55.341432 (+ 290us) server_negotiation.cc:299] Negotiation successful
0925 04:48:55.341838 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":146,"threads_started":1}
I20260925 04:48:55.345118 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:55.335374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:42903 (local address 127.0.0.1:42698)
0925 04:48:55.335745 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:55.335804 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 04:48:55.335929 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:48:55.336234 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:48:55.336238 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:48:55.336571 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:48:55.336762 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:55.336768 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:55.337626 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:55.337632 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:55.338497 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:55.338508 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:55.339124 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:55.339128 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:55.339223 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:55.339750 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:48:55.339766 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:48:55.341417 (+ 1651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:48:55.343330 (+ 1913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:48:55.343337 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:48:55.343342 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:48:55.343619 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:48:55.343984 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:48:55.343987 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:48:55.343988 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:48:55.344063 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:48:55.344508 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:48:55.344513 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:48:55.344610 (+ 97us) client_negotiation.cc:769] Sending connection context
0925 04:48:55.344780 (+ 170us) client_negotiation.cc:240] Negotiation successful
0925 04:48:55.344958 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":121,"threads_started":1}
I20260925 04:48:55.345631 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:55.335482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42698 (local address 127.0.98.126:42903)
0925 04:48:55.335663 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:55.335668 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:55.335898 (+ 230us) server_negotiation.cc:407] Connection header received
0925 04:48:55.336025 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:55.336028 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:55.336091 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:55.336184 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:55.336903 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:55.337507 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:55.338631 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:55.338974 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:55.339110 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:55.341578 (+ 2468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:55.341585 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:55.341592 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:48:55.341626 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:55.343227 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:55.343770 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:55.343774 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:55.343776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:55.343879 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:55.344156 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:55.344159 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:55.344161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:55.344379 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:55.344502 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:55.345124 (+ 622us) server_negotiation.cc:299] Negotiation successful
0925 04:48:55.345484 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260925 04:48:55.346671 23257 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.125" port: 36771 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 04:48:55 dist-test-slave-kghz krb5kdc[23024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
Sep 25 04:48:55 dist-test-slave-kghz krb5kdc[23024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311735, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.124@KRBTEST.COM for kudu/127.0.98.125@KRBTEST.COM
I20260925 04:48:55.354136 23105 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.124" port: 40191 } attempt: 3
I20260925 04:48:55.355726 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:55.345589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42702 (local address 127.0.98.126:42903)
0925 04:48:55.345812 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:55.345817 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:55.345832 (+ 15us) server_negotiation.cc:407] Connection header received
0925 04:48:55.345870 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:55.345873 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:55.345917 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:55.345994 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:55.346754 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:55.347474 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:55.348474 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:55.348798 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:55.348911 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:55.351783 (+ 2872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:55.351786 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:55.351788 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:48:55.351812 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:55.353353 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:55.353801 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:55.353807 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:55.353810 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:55.353921 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:55.354296 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:55.354299 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:55.354301 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:55.354553 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:55.355074 (+ 521us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:55.355413 (+ 339us) server_negotiation.cc:299] Negotiation successful
0925 04:48:55.355534 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":170,"threads_started":1}
I20260925 04:48:55.356451 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:55.347179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:36771 (local address 127.0.0.1:51802)
0925 04:48:55.347319 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:55.347334 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:48:55.347451 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:48:55.347689 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:48:55.347692 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:48:55.347862 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:48:55.347972 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:55.347978 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:55.349008 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:55.349012 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:55.349852 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:55.349867 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:55.350509 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:55.350515 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:55.350649 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:55.351164 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:48:55.351184 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:48:55.352456 (+ 1272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:48:55.354206 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:48:55.354212 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:48:55.354214 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:48:55.354505 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:48:55.354829 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:48:55.354833 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:48:55.354835 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:48:55.354913 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:48:55.355893 (+ 980us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:48:55.355897 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:48:55.355999 (+ 102us) client_negotiation.cc:769] Sending connection context
0925 04:48:55.356124 (+ 125us) client_negotiation.cc:240] Negotiation successful
0925 04:48:55.356252 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260925 04:48:55.356601 23185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:55.347286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51802 (local address 127.0.98.125:36771)
0925 04:48:55.347413 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:55.347417 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:55.347433 (+ 16us) server_negotiation.cc:407] Connection header received
0925 04:48:55.347523 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:55.347529 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:55.347587 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:55.347661 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:55.348119 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:55.348894 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:55.350003 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:55.350366 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:55.350477 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:55.352597 (+ 2120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:55.352599 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:55.352601 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:48:55.352652 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:55.354086 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:55.354620 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:55.354624 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:55.354626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:55.354714 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:55.355009 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:55.355012 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:55.355014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:55.355247 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:55.355370 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:55.356200 (+ 830us) server_negotiation.cc:299] Negotiation successful
0925 04:48:55.356327 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 04:48:55.357340 23257 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.124" port: 40191 } has no permanent_uuid. Determining permanent_uuid...
I20260925 04:48:55.358192 23180 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.124" port: 40191 } attempt: 3
Sep 25 04:48:55 dist-test-slave-kghz krb5kdc[23024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311734, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for kudu/127.0.98.124@KRBTEST.COM
Sep 25 04:48:55 dist-test-slave-kghz krb5kdc[23024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.125@KRBTEST.COM
I20260925 04:48:55.364655 23108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:55.354636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:40191 (local address 127.0.0.1:48962)
0925 04:48:55.354759 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:55.354774 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:48:55.354842 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:48:55.355094 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:48:55.355098 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:48:55.355271 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:48:55.355398 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:55.355405 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:55.357003 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:55.357008 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:55.357912 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:55.357923 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:55.358571 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:55.358575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:55.358661 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:55.359151 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:48:55.359166 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:48:55.360638 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:48:55.362876 (+ 2238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:48:55.362880 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:48:55.362882 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:48:55.363167 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:48:55.363473 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:48:55.363476 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:48:55.363479 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:48:55.363573 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:48:55.364079 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:48:55.364083 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:48:55.364211 (+ 128us) client_negotiation.cc:769] Sending connection context
0925 04:48:55.364305 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 04:48:55.364444 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
Sep 25 04:48:55 dist-test-slave-kghz krb5kdc[23024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311735, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.124@KRBTEST.COM for kudu/127.0.98.124@KRBTEST.COM
I20260925 04:48:55.365353 23260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:55.354691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48962 (local address 127.0.98.124:40191)
0925 04:48:55.354821 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:55.354825 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:55.354837 (+ 12us) server_negotiation.cc:407] Connection header received
0925 04:48:55.354939 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:55.354942 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:55.354983 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:55.355058 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:55.355517 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:55.356575 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:55.358043 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:55.358441 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:55.358883 (+ 442us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:55.360792 (+ 1909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:55.360796 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:55.360798 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:48:55.360821 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:55.362754 (+ 1933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:55.363274 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:55.363277 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:55.363279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:55.363362 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:55.363714 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:55.363717 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:55.363719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:55.363962 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:55.365014 (+ 1052us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:55.365085 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 04:48:55.365208 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Sep 25 04:48:55 dist-test-slave-kghz krb5kdc[23024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311735, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.125@KRBTEST.COM for kudu/127.0.98.124@KRBTEST.COM
I20260925 04:48:55.369247 23105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1268f111a3bd40469d44908cf343a0db: Bootstrap starting.
I20260925 04:48:55.369421 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:55.357862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:40191 (local address 127.0.0.1:48974)
0925 04:48:55.358044 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:55.358059 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:48:55.358127 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:48:55.358759 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:48:55.358766 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:48:55.358959 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:48:55.359066 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:55.359076 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:55.360668 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:55.360671 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:55.361584 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:55.361595 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:55.363113 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:55.363120 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:55.363209 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:55.363699 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:48:55.363713 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:48:55.365069 (+ 1356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:48:55.367841 (+ 2772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:48:55.367844 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:48:55.367846 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:48:55.368122 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:48:55.368469 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:48:55.368471 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:48:55.368473 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:48:55.368553 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:48:55.368941 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:48:55.368943 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:48:55.369017 (+ 74us) client_negotiation.cc:769] Sending connection context
0925 04:48:55.369104 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 04:48:55.369238 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"mutex_wait_us":62}
I20260925 04:48:55.369944 23263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:55.357903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48974 (local address 127.0.98.124:40191)
0925 04:48:55.358529 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:55.358538 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:55.358551 (+ 13us) server_negotiation.cc:407] Connection header received
0925 04:48:55.358582 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:55.358585 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:55.358627 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:55.359174 (+ 547us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:55.359265 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:55.360567 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:55.361689 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:55.362995 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:55.363458 (+ 463us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:55.365211 (+ 1753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:55.365215 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:55.365218 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:48:55.365245 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:55.367734 (+ 2489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:55.368220 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:55.368223 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:55.368225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:55.368364 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:55.368642 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:55.368644 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:55.368646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:55.368844 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:55.369576 (+ 732us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:55.369674 (+ 98us) server_negotiation.cc:299] Negotiation successful
0925 04:48:55.369788 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":117,"threads_started":1}
I20260925 04:48:55.370286 23185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:55.356931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51812 (local address 127.0.98.125:36771)
0925 04:48:55.357044 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:55.357048 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:55.357059 (+ 11us) server_negotiation.cc:407] Connection header received
0925 04:48:55.357163 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:55.357166 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:55.357219 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:55.357298 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:55.358047 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:55.358786 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:55.359918 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:55.360259 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:55.360347 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:55.363702 (+ 3355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:55.363707 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:55.363713 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:48:55.363745 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:55.367433 (+ 3688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:55.368004 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:55.368010 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:55.368015 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:55.368105 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:55.368500 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:55.368504 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:55.368506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:55.369064 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:55.369277 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:55.369976 (+ 699us) server_negotiation.cc:299] Negotiation successful
0925 04:48:55.370090 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":63,"spinlock_wait_cycles":3840}
I20260925 04:48:55.372526 23105 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1268f111a3bd40469d44908cf343a0db: Neither blocks nor log segments found. Creating new log.
I20260925 04:48:55.373487 23105 log.cc:824] T 00000000000000000000000000000000 P 1268f111a3bd40469d44908cf343a0db: Log is configured to *not* fsync() on all Append() calls
I20260925 04:48:55.373862 23257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd77afae8b7d417a8896d0f57f48ea2d: Bootstrap starting.
I20260925 04:48:55.374505 393 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:48:55.376284 23105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1268f111a3bd40469d44908cf343a0db: No bootstrap required, opened a new log
I20260925 04:48:55.376709 23257 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd77afae8b7d417a8896d0f57f48ea2d: Neither blocks nor log segments found. Creating new log.
I20260925 04:48:55.377568 23257 log.cc:824] T 00000000000000000000000000000000 P cd77afae8b7d417a8896d0f57f48ea2d: Log is configured to *not* fsync() on all Append() calls
I20260925 04:48:55.378203 23189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:55.358575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:40191 (local address 127.0.0.1:48982)
0925 04:48:55.358738 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 04:48:55.358751 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:48:55.359183 (+ 432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:48:55.359870 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:48:55.359873 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:48:55.360066 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:48:55.360194 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:55.360200 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:55.362041 (+ 1841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:55.362044 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:55.363002 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:48:55.363018 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:48:55.363747 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:48:55.363750 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:48:55.363859 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:55.364304 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:48:55.364321 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:48:55.365907 (+ 1586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:48:55.368500 (+ 2593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:48:55.368504 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:48:55.368506 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:48:55.368739 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:48:55.369027 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:48:55.369030 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:48:55.369032 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:48:55.369112 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:48:55.369899 (+ 787us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:48:55.369905 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:48:55.370002 (+ 97us) client_negotiation.cc:769] Sending connection context
0925 04:48:55.370111 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 04:48:55.370227 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"spinlock_wait_cycles":2176}
I20260925 04:48:55.378425 23079 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "1268f111a3bd40469d44908cf343a0db"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42702
I20260925 04:48:55.379036 23264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:55.359320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48982 (local address 127.0.98.124:40191)
0925 04:48:55.359613 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:55.359621 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:55.359637 (+ 16us) server_negotiation.cc:407] Connection header received
0925 04:48:55.359673 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:55.359677 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:55.359722 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:55.360297 (+ 575us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:55.360428 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:55.361929 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:55.363206 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:55.363627 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:55.364065 (+ 438us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:55.366043 (+ 1978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:55.366048 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:55.366050 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:48:55.366077 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:55.368385 (+ 2308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:55.368842 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:55.368846 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:55.368847 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:55.368923 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:55.369223 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:55.369226 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:55.369227 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:55.369775 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:55.378566 (+ 8791us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:55.378676 (+ 110us) server_negotiation.cc:299] Negotiation successful
0925 04:48:55.378834 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":222,"thread_start_us":113,"threads_started":1}
I20260925 04:48:55.380455 23257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd77afae8b7d417a8896d0f57f48ea2d: No bootstrap required, opened a new log
I20260925 04:48:55.381189 23105 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1268f111a3bd40469d44908cf343a0db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1268f111a3bd40469d44908cf343a0db" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42903 } } peers { permanent_uuid: "b2d4802490b74daf939925e65397742c" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 36771 } } peers { permanent_uuid: "cd77afae8b7d417a8896d0f57f48ea2d" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 40191 } }
I20260925 04:48:55.381412 23105 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1268f111a3bd40469d44908cf343a0db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1268f111a3bd40469d44908cf343a0db, State: Initialized, Role: FOLLOWER
I20260925 04:48:55.381745 23105 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1268f111a3bd40469d44908cf343a0db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1268f111a3bd40469d44908cf343a0db" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42903 } } peers { permanent_uuid: "b2d4802490b74daf939925e65397742c" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 36771 } } peers { permanent_uuid: "cd77afae8b7d417a8896d0f57f48ea2d" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 40191 } }
I20260925 04:48:55.382102 23180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2d4802490b74daf939925e65397742c: Bootstrap starting.
I20260925 04:48:55.382575 23267 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1268f111a3bd40469d44908cf343a0db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1268f111a3bd40469d44908cf343a0db" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42903 } } peers { permanent_uuid: "b2d4802490b74daf939925e65397742c" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 36771 } } peers { permanent_uuid: "cd77afae8b7d417a8896d0f57f48ea2d" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 40191 } } }
I20260925 04:48:55.382893 23267 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1268f111a3bd40469d44908cf343a0db [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:48:55.383152 23105 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1268f111a3bd40469d44908cf343a0db [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:48:55.383206 23079 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1268f111a3bd40469d44908cf343a0db [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 04:48:55.383324 23079 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1268f111a3bd40469d44908cf343a0db [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:48:55.384719 23079 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1268f111a3bd40469d44908cf343a0db [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1268f111a3bd40469d44908cf343a0db" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42903 } } peers { permanent_uuid: "b2d4802490b74daf939925e65397742c" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 36771 } } peers { permanent_uuid: "cd77afae8b7d417a8896d0f57f48ea2d" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 40191 } }
I20260925 04:48:55.384933 23180 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2d4802490b74daf939925e65397742c: Neither blocks nor log segments found. Creating new log.
I20260925 04:48:55.385169 23257 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd77afae8b7d417a8896d0f57f48ea2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1268f111a3bd40469d44908cf343a0db" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42903 } } peers { permanent_uuid: "b2d4802490b74daf939925e65397742c" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 36771 } } peers { permanent_uuid: "cd77afae8b7d417a8896d0f57f48ea2d" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 40191 } }
I20260925 04:48:55.385417 23257 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd77afae8b7d417a8896d0f57f48ea2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd77afae8b7d417a8896d0f57f48ea2d, State: Initialized, Role: FOLLOWER
I20260925 04:48:55.385669 23079 leader_election.cc:288] T 00000000000000000000000000000000 P 1268f111a3bd40469d44908cf343a0db [CANDIDATE]: Term 1 election: Requested vote from peers b2d4802490b74daf939925e65397742c (127.0.98.125:36771), cd77afae8b7d417a8896d0f57f48ea2d (127.0.98.124:40191)
I20260925 04:48:55.385731 23257 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd77afae8b7d417a8896d0f57f48ea2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1268f111a3bd40469d44908cf343a0db" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42903 } } peers { permanent_uuid: "b2d4802490b74daf939925e65397742c" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 36771 } } peers { permanent_uuid: "cd77afae8b7d417a8896d0f57f48ea2d" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 40191 } }
I20260925 04:48:55.385963 23180 log.cc:824] T 00000000000000000000000000000000 P b2d4802490b74daf939925e65397742c: Log is configured to *not* fsync() on all Append() calls
I20260925 04:48:55.387161 23257 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd77afae8b7d417a8896d0f57f48ea2d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:48:55.387226 23229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1268f111a3bd40469d44908cf343a0db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "cd77afae8b7d417a8896d0f57f48ea2d"
I20260925 04:48:55.387439 23229 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd77afae8b7d417a8896d0f57f48ea2d [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:48:55.388149 23180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2d4802490b74daf939925e65397742c: No bootstrap required, opened a new log
I20260925 04:48:55.388765 23229 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cd77afae8b7d417a8896d0f57f48ea2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1268f111a3bd40469d44908cf343a0db in term 1.
I20260925 04:48:55.389472 23055 leader_election.cc:302] T 00000000000000000000000000000000 P 1268f111a3bd40469d44908cf343a0db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1268f111a3bd40469d44908cf343a0db, cd77afae8b7d417a8896d0f57f48ea2d; no voters:
I20260925 04:48:55.390210 23267 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1268f111a3bd40469d44908cf343a0db [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:48:55.390439 23267 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1268f111a3bd40469d44908cf343a0db [term 1 LEADER]: Becoming Leader. State: Replica: 1268f111a3bd40469d44908cf343a0db, State: Running, Role: LEADER
I20260925 04:48:55.390799 23267 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1268f111a3bd40469d44908cf343a0db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1268f111a3bd40469d44908cf343a0db" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42903 } } peers { permanent_uuid: "b2d4802490b74daf939925e65397742c" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 36771 } } peers { permanent_uuid: "cd77afae8b7d417a8896d0f57f48ea2d" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 40191 } }
I20260925 04:48:55.392719 23267 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1268f111a3bd40469d44908cf343a0db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1268f111a3bd40469d44908cf343a0db. Latest consensus state: current_term: 1 leader_uuid: "1268f111a3bd40469d44908cf343a0db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1268f111a3bd40469d44908cf343a0db" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42903 } } peers { permanent_uuid: "b2d4802490b74daf939925e65397742c" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 36771 } } peers { permanent_uuid: "cd77afae8b7d417a8896d0f57f48ea2d" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 40191 } } }
I20260925 04:48:55.392848 23267 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1268f111a3bd40469d44908cf343a0db [sys.catalog]: This master's current role is: LEADER
I20260925 04:48:55.393150 23180 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2d4802490b74daf939925e65397742c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1268f111a3bd40469d44908cf343a0db" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42903 } } peers { permanent_uuid: "b2d4802490b74daf939925e65397742c" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 36771 } } peers { permanent_uuid: "cd77afae8b7d417a8896d0f57f48ea2d" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 40191 } }
I20260925 04:48:55.393383 23180 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2d4802490b74daf939925e65397742c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2d4802490b74daf939925e65397742c, State: Initialized, Role: FOLLOWER
I20260925 04:48:55.393731 23180 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2d4802490b74daf939925e65397742c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1268f111a3bd40469d44908cf343a0db" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42903 } } peers { permanent_uuid: "b2d4802490b74daf939925e65397742c" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 36771 } } peers { permanent_uuid: "cd77afae8b7d417a8896d0f57f48ea2d" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 40191 } }
I20260925 04:48:55.391781 23037 master_runner.cc:186] Error getting master registration for 127.0.98.125:36771: OK, instance_id { permanent_uuid: "b2d4802490b74daf939925e65397742c" instance_seqno: 1790311735100865 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 04:48:55.395165 23180 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2d4802490b74daf939925e65397742c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:48:55.395277 23037 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260925 04:48:55.395351 23278 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1268f111a3bd40469d44908cf343a0db: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 04:48:55.395419 23278 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 04:48:55.395619 23154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1268f111a3bd40469d44908cf343a0db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b2d4802490b74daf939925e65397742c"
I20260925 04:48:55.396401 23300 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2d4802490b74daf939925e65397742c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1268f111a3bd40469d44908cf343a0db" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42903 } } peers { permanent_uuid: "b2d4802490b74daf939925e65397742c" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 36771 } } peers { permanent_uuid: "cd77afae8b7d417a8896d0f57f48ea2d" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 40191 } } }
I20260925 04:48:55.396531 23300 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2d4802490b74daf939925e65397742c [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:48:55.396529 23154 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2d4802490b74daf939925e65397742c [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:48:55.397908 23154 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b2d4802490b74daf939925e65397742c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1268f111a3bd40469d44908cf343a0db in term 1.
I20260925 04:48:55.398380 23187 master_runner.cc:186] Error getting master registration for 127.0.98.125:36771: OK, instance_id { permanent_uuid: "b2d4802490b74daf939925e65397742c" instance_seqno: 1790311735100865 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 04:48:55.401914 23187 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 04:48:55.401619 23279 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd77afae8b7d417a8896d0f57f48ea2d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1268f111a3bd40469d44908cf343a0db" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42903 } } peers { permanent_uuid: "b2d4802490b74daf939925e65397742c" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 36771 } } peers { permanent_uuid: "cd77afae8b7d417a8896d0f57f48ea2d" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 40191 } } }
I20260925 04:48:55.402495 23279 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd77afae8b7d417a8896d0f57f48ea2d [sys.catalog]: This master's current role is: FOLLOWER
W20260925 04:48:55.402539 23311 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b2d4802490b74daf939925e65397742c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 04:48:55.402604 23311 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 04:48:55.412773 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:55.398027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42712 (local address 127.0.98.126:42903)
0925 04:48:55.398147 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:55.398153 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:55.402835 (+ 4682us) server_negotiation.cc:407] Connection header received
0925 04:48:55.403027 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:55.403030 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:55.403078 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:55.403127 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:55.404157 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:55.404931 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:55.406216 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:55.406605 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:55.406690 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:55.408903 (+ 2213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:55.408907 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:55.408911 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:48:55.408940 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:55.410705 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:55.411234 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:55.411238 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:55.411240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:55.411331 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:55.411638 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:55.411641 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:55.411643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:55.411878 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:55.411997 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:55.412385 (+ 388us) server_negotiation.cc:299] Negotiation successful
0925 04:48:55.412554 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 04:48:55.416778 23299 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260925 04:48:55.416872 23295 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cd77afae8b7d417a8896d0f57f48ea2d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 04:48:55.416949 23295 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 04:48:55.417740 23299 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:48:55.429535 23185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:55.401838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51814 (local address 127.0.98.125:36771)
0925 04:48:55.401982 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:55.401988 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:55.404329 (+ 2341us) server_negotiation.cc:407] Connection header received
0925 04:48:55.404509 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:55.404512 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:55.404558 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:55.404607 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:55.405567 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:55.406395 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:55.407747 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:55.408116 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:55.408191 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:55.422997 (+ 14806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:55.423002 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:55.423005 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:48:55.423040 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:55.424769 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:55.427620 (+ 2851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:55.427624 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:55.427626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:55.427722 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:55.428019 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:55.428023 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:55.428024 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:55.428259 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:55.428370 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:55.429199 (+ 829us) server_negotiation.cc:299] Negotiation successful
0925 04:48:55.429343 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 04:48:55.430923 23229 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cd77afae8b7d417a8896d0f57f48ea2d [term 1 FOLLOWER]: Refusing update from remote peer 1268f111a3bd40469d44908cf343a0db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:48:55.431640 23296 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1268f111a3bd40469d44908cf343a0db [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd77afae8b7d417a8896d0f57f48ea2d" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 40191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:48:55.432533 23264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:55.402184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48984 (local address 127.0.98.124:40191)
0925 04:48:55.402326 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:55.402330 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:55.409079 (+ 6749us) server_negotiation.cc:407] Connection header received
0925 04:48:55.409573 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:55.409577 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:55.409637 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:55.409693 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:55.410961 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:55.411812 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:55.413072 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:55.413456 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:55.413553 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:55.425323 (+ 11770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:48:55.425327 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:48:55.425330 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:48:55.425363 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:48:55.427095 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:55.428704 (+ 1609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:55.428707 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:55.428709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:55.428802 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:48:55.431382 (+ 2580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:48:55.431387 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:48:55.431389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:48:55.431642 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:48:55.431771 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:55.432216 (+ 445us) server_negotiation.cc:299] Negotiation successful
0925 04:48:55.432377 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 04:48:55.436288 23279 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd77afae8b7d417a8896d0f57f48ea2d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1268f111a3bd40469d44908cf343a0db. Latest consensus state: current_term: 1 leader_uuid: "1268f111a3bd40469d44908cf343a0db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1268f111a3bd40469d44908cf343a0db" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42903 } } peers { permanent_uuid: "b2d4802490b74daf939925e65397742c" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 36771 } } peers { permanent_uuid: "cd77afae8b7d417a8896d0f57f48ea2d" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 40191 } } }
I20260925 04:48:55.436447 23279 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd77afae8b7d417a8896d0f57f48ea2d [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:48:55.438274 23154 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b2d4802490b74daf939925e65397742c [term 1 FOLLOWER]: Refusing update from remote peer 1268f111a3bd40469d44908cf343a0db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:48:55.438925 23296 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1268f111a3bd40469d44908cf343a0db [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2d4802490b74daf939925e65397742c" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 36771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:48:55.443732 23300 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2d4802490b74daf939925e65397742c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1268f111a3bd40469d44908cf343a0db. Latest consensus state: current_term: 1 leader_uuid: "1268f111a3bd40469d44908cf343a0db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1268f111a3bd40469d44908cf343a0db" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42903 } } peers { permanent_uuid: "b2d4802490b74daf939925e65397742c" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 36771 } } peers { permanent_uuid: "cd77afae8b7d417a8896d0f57f48ea2d" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 40191 } } }
I20260925 04:48:55.443907 23300 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2d4802490b74daf939925e65397742c [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:48:55.451184 23296 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1268f111a3bd40469d44908cf343a0db [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1268f111a3bd40469d44908cf343a0db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1268f111a3bd40469d44908cf343a0db" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42903 } } peers { permanent_uuid: "b2d4802490b74daf939925e65397742c" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 36771 } } peers { permanent_uuid: "cd77afae8b7d417a8896d0f57f48ea2d" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 40191 } } }
I20260925 04:48:55.451344 23296 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1268f111a3bd40469d44908cf343a0db [sys.catalog]: This master's current role is: LEADER
I20260925 04:48:55.451463 23317 mvcc.cc:205] Tried to move back new op lower bound from 7333116868314730496 to 7333116868164321280. Current Snapshot: MvccSnapshot[applied={T|T < 7333116868314730496}]
I20260925 04:48:55.452925 23279 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd77afae8b7d417a8896d0f57f48ea2d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1268f111a3bd40469d44908cf343a0db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1268f111a3bd40469d44908cf343a0db" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42903 } } peers { permanent_uuid: "b2d4802490b74daf939925e65397742c" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 36771 } } peers { permanent_uuid: "cd77afae8b7d417a8896d0f57f48ea2d" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 40191 } } }
I20260925 04:48:55.453063 23279 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd77afae8b7d417a8896d0f57f48ea2d [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:48:55.460083 23299 catalog_manager.cc:1347] Generated new cluster ID: c0491a6254f2451083d088af548ca144
I20260925 04:48:55.460163 23299 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:48:55.466958 23267 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1268f111a3bd40469d44908cf343a0db [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1268f111a3bd40469d44908cf343a0db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1268f111a3bd40469d44908cf343a0db" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42903 } } peers { permanent_uuid: "b2d4802490b74daf939925e65397742c" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 36771 } } peers { permanent_uuid: "cd77afae8b7d417a8896d0f57f48ea2d" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 40191 } } }
I20260925 04:48:55.467113 23267 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1268f111a3bd40469d44908cf343a0db [sys.catalog]: This master's current role is: LEADER
I20260925 04:48:55.468250 23300 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2d4802490b74daf939925e65397742c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1268f111a3bd40469d44908cf343a0db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1268f111a3bd40469d44908cf343a0db" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42903 } } peers { permanent_uuid: "b2d4802490b74daf939925e65397742c" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 36771 } } peers { permanent_uuid: "cd77afae8b7d417a8896d0f57f48ea2d" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 40191 } } }
I20260925 04:48:55.468382 23300 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2d4802490b74daf939925e65397742c [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:48:55.511049 23299 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:48:55.512403 23299 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:48:55.529500 23299 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1268f111a3bd40469d44908cf343a0db: Generated new TSK 0
I20260925 04:48:55.530418 23299 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 04:48:55.558579 23109 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:48:55.555436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42718 (local address 127.0.98.126:42903)
0925 04:48:55.555577 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:55.555582 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:55.557375 (+ 1793us) server_negotiation.cc:407] Connection header received
0925 04:48:55.557634 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:55.557638 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:55.557693 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:55.557746 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:55.558441 (+ 695us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42718: BlockingRecv error: recv got EOF from 127.0.0.1:42718 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260925 04:48:55.560316 23185 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:48:55.556181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51820 (local address 127.0.98.125:36771)
0925 04:48:55.556301 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:55.556306 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:55.558971 (+ 2665us) server_negotiation.cc:407] Connection header received
0925 04:48:55.559321 (+ 350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:55.559324 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:55.559383 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:55.559446 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:55.560207 (+ 761us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51820: BlockingRecv error: recv got EOF from 127.0.0.1:51820 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260925 04:48:55.561374 23264 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:48:55.556776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48998 (local address 127.0.98.124:40191)
0925 04:48:55.556885 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:55.556890 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:55.559688 (+ 2798us) server_negotiation.cc:407] Connection header received
0925 04:48:55.560645 (+ 957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:55.560649 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:55.560700 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:55.560751 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:55.561257 (+ 506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48998: BlockingRecv error: recv got EOF from 127.0.0.1:48998 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260925 04:48:55.565532 23185 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:48:55.563922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51836 (local address 127.0.98.125:36771)
0925 04:48:55.564044 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:55.564049 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:55.564529 (+ 480us) server_negotiation.cc:407] Connection header received
0925 04:48:55.564744 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:55.564747 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:55.564840 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:55.564905 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:55.565425 (+ 520us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51836: BlockingRecv error: recv got EOF from 127.0.0.1:51836 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260925 04:48:55.572386 23185 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:48:55.570638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51838 (local address 127.0.98.125:36771)
0925 04:48:55.570752 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:55.570756 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:55.571358 (+ 602us) server_negotiation.cc:407] Connection header received
0925 04:48:55.571572 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:55.571576 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:55.571633 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:55.571689 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:55.572272 (+ 583us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51838: BlockingRecv error: recv got EOF from 127.0.0.1:51838 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260925 04:48:55.578203 23185 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:48:55.576525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51842 (local address 127.0.98.125:36771)
0925 04:48:55.576637 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:55.576641 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:55.577215 (+ 574us) server_negotiation.cc:407] Connection header received
0925 04:48:55.577415 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:55.577418 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:55.577472 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:55.577531 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:55.578095 (+ 564us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51842: BlockingRecv error: recv got EOF from 127.0.0.1:51842 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260925 04:48:55.586539 23185 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:48:55.584699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51858 (local address 127.0.98.125:36771)
0925 04:48:55.584848 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:55.584853 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:55.585532 (+ 679us) server_negotiation.cc:407] Connection header received
0925 04:48:55.585741 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:55.585744 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:55.585799 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:55.585855 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:55.586425 (+ 570us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51858: BlockingRecv error: recv got EOF from 127.0.0.1:51858 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260925 04:48:55.599558 23185 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:48:55.596882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51860 (local address 127.0.98.125:36771)
0925 04:48:55.597012 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:55.597018 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:55.598503 (+ 1485us) server_negotiation.cc:407] Connection header received
0925 04:48:55.598706 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:55.598710 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:55.598767 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:55.598822 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:55.599434 (+ 612us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51860: BlockingRecv error: recv got EOF from 127.0.0.1:51860 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260925 04:48:55.619388 23185 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:48:55.618042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51864 (local address 127.0.98.125:36771)
0925 04:48:55.618176 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:55.618181 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:55.618470 (+ 289us) server_negotiation.cc:407] Connection header received
0925 04:48:55.618646 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:55.618649 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:55.618708 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:55.618804 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:55.619264 (+ 460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51864: BlockingRecv error: recv got EOF from 127.0.0.1:51864 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260925 04:48:55.654749 23185 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:48:55.653495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51870 (local address 127.0.98.125:36771)
0925 04:48:55.653656 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:55.653662 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:55.653779 (+ 117us) server_negotiation.cc:407] Connection header received
0925 04:48:55.653955 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:55.653959 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:55.654016 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:55.654109 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:55.654638 (+ 529us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51870: BlockingRecv error: recv got EOF from 127.0.0.1:51870 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
W20260925 04:48:55.722398 23185 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:48:55.721070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51880 (local address 127.0.98.125:36771)
0925 04:48:55.721238 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:55.721243 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:55.721387 (+ 144us) server_negotiation.cc:407] Connection header received
0925 04:48:55.721591 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:55.721594 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:55.721641 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:55.721734 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:55.722217 (+ 483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51880: BlockingRecv error: recv got EOF from 127.0.0.1:51880 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260925 04:48:55.853938 23185 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:48:55.852583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51896 (local address 127.0.98.125:36771)
0925 04:48:55.852810 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:55.852815 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:55.852956 (+ 141us) server_negotiation.cc:407] Connection header received
0925 04:48:55.853154 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:55.853158 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:55.853206 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:55.853300 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:55.853766 (+ 466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51896: BlockingRecv error: recv got EOF from 127.0.0.1:51896 (error 108)
Metrics: {"server-negotiator.queue_time_us":125}
W20260925 04:48:56.113555 23185 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:48:56.112317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51904 (local address 127.0.98.125:36771)
0925 04:48:56.112457 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:56.112462 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:56.112585 (+ 123us) server_negotiation.cc:407] Connection header received
0925 04:48:56.112737 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:56.112740 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:56.112792 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:56.112875 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:48:56.113432 (+ 557us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51904: BlockingRecv error: recv got EOF from 127.0.0.1:51904 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 04:48:56.403829 23311 catalog_manager.cc:1259] Loaded cluster ID: c0491a6254f2451083d088af548ca144
I20260925 04:48:56.403913 23311 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b2d4802490b74daf939925e65397742c: loading cluster ID for follower catalog manager: success
I20260925 04:48:56.405506 23311 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b2d4802490b74daf939925e65397742c: acquiring CA information for follower catalog manager: success
I20260925 04:48:56.406698 23311 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b2d4802490b74daf939925e65397742c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 04:48:56.418174 23295 catalog_manager.cc:1259] Loaded cluster ID: c0491a6254f2451083d088af548ca144
I20260925 04:48:56.418319 23295 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cd77afae8b7d417a8896d0f57f48ea2d: loading cluster ID for follower catalog manager: success
I20260925 04:48:56.419993 23295 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cd77afae8b7d417a8896d0f57f48ea2d: acquiring CA information for follower catalog manager: success
I20260925 04:48:56.421129 23295 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cd77afae8b7d417a8896d0f57f48ea2d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 04:48:56.631702 23389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:56.627791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51908 (local address 127.0.98.125:36771)
0925 04:48:56.628095 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:56.628099 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:56.628118 (+ 19us) server_negotiation.cc:407] Connection header received
0925 04:48:56.628209 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:56.628212 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:56.628334 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:56.628438 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:48:56.628714 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:56.629274 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:56.630217 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:56.630485 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:56.630572 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:56.630924 (+ 352us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:48:56.631358 (+ 434us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:48:56.631469 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:56.631526 (+ 57us) server_negotiation.cc:299] Negotiation successful
0925 04:48:56.631575 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":120,"threads_started":1}
I20260925 04:48:56.639216 23395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:56.635154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49000 (local address 127.0.98.124:40191)
0925 04:48:56.635409 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:56.635412 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:56.635425 (+ 13us) server_negotiation.cc:407] Connection header received
0925 04:48:56.635502 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:56.635504 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:56.635618 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:56.635718 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:48:56.635953 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:56.636492 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:56.637393 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:56.637626 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:56.637720 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:56.638092 (+ 372us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:48:56.638674 (+ 582us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:48:56.638798 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:56.638923 (+ 125us) server_negotiation.cc:299] Negotiation successful
0925 04:48:56.638989 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":74,"threads_started":1}
I20260925 04:48:56.647107 23395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:56.643234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49004 (local address 127.0.98.124:40191)
0925 04:48:56.643353 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:56.643360 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:56.643608 (+ 248us) server_negotiation.cc:407] Connection header received
0925 04:48:56.643758 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:56.643761 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:56.643810 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:56.643897 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:48:56.644185 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:56.644854 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:56.645821 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:56.646136 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:56.646252 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:56.646605 (+ 353us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:48:56.646688 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:48:56.646783 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:56.646859 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 04:48:56.646927 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 04:48:56.647257 23389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:56.642927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51922 (local address 127.0.98.125:36771)
0925 04:48:56.643090 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:56.643095 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:56.643216 (+ 121us) server_negotiation.cc:407] Connection header received
0925 04:48:56.643579 (+ 363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:56.643582 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:56.643648 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:56.643718 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:48:56.644055 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:56.644860 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:56.646017 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:56.646388 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:56.646495 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:56.646778 (+ 283us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:48:56.646859 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:48:56.646925 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:56.647067 (+ 142us) server_negotiation.cc:299] Negotiation successful
0925 04:48:56.647125 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 04:48:56.647701 23401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:48:56.642477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42730 (local address 127.0.98.126:42903)
0925 04:48:56.642728 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0925 04:48:56.642735 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:48:56.642767 (+ 32us) server_negotiation.cc:407] Connection header received
0925 04:48:56.642968 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:48:56.642972 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:48:56.643143 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:48:56.643268 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:48:56.643675 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:56.644511 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:56.645745 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:48:56.646114 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:48:56.646216 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:48:56.646682 (+ 466us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:48:56.647073 (+ 391us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:48:56.647157 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 04:48:56.647447 (+ 290us) server_negotiation.cc:299] Negotiation successful
0925 04:48:56.647522 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":67,"threads_started":1}
I20260925 04:48:56.650854 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 23037
I20260925 04:48:56.657483 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 23112
I20260925 04:48:56.663436 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHechoP/build/debug/bin/kudu with pid 23187
2026-09-25T04:48:56Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3129 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6300 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (286777 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST